ZKPodcast: Decentralised Storage with Arweave

Transcript

[Music] welcome to zero knowledge a podcast where we explore the latest in blockchain technology and the decentralized web the show is hosted by me Anna and me Fredrik [Music] in this episode we sit down with Sam Williams from our we've to talk about decentralized storage and how they address some of the problems with that in their software [Music] so today Frederick and I are sitting

here with Sam Williams hi Sam hi thanks for having me Sam's from our weave today we're gonna be talking a little bit about decentralized storage and the decentralized storage topic so this is something that we kind of started talking about over a year ago we had a decentralized storage series still not finished we just we've decided we're not gonna make it a three partners we had originally thought but rather just have it be a continuum but I think today we will hopefully get a chance to speak a little bit more about this topic and maybe find out a little bit of how our weave plans on tackling it so maybe to start off do you want to just introduce yourself hi I'm Sam Williams I'm the CEO

and one of the co-founders the our weave project the our weave project is essentially a permanent information storage system so we've taken a blockchain and modified it so that you can essentially store very large amounts of data on the chain itself and it has a sort of crypto economic incentive ization mechanism that allows you to persist that data for very long periods of time cool so let's dig into that a little bit so I think this is super interesting you know how you actually write this blockchain and how you incentivize things and how you ensure availability and everything else I mean there's a ton of problems here to unpack and this is something that a ton of projects are trying to tackle so maybe like start out at the core of you know

what what is unique to our weave and what is it built around yeah I mean the thing is built around is this data structure that we call a block weave it's essentially a modified blockchain where instead of just including the hash of the last block in the production of the next block you also pick a pseudo randomly chosen previous block from the network and you require that in the proof-of-work to produce the next block in the Syst and so what this means because the miners can't essentially guess ahead of time which block is going to be required to mine say block you know n plus one is they have to store as many of the old blocks as possible and the consequence of this is what we call proof of access so they need to be able to prove that they can access the old blocks in the

system in order to produce the new blocks and because storage is exceptionally cheap in comparison to hashing they're incentivized to optimize their storage capacity before they optimize their hashing capabilities and so this essentially means is really strong incentive for miners to replicate as much of the old data structure in the network as possible basically you're using this pseudo randomness to make sure that a miner keeps the chain but is this also like are you also storing data in that yes exactly that okay and the miner we've essentially memorized the state of the blockchain so you can join the network by just downloading I said it about 50 bucks but you can obviously download the whole thing if you want to verify it

yeah but the expectation really is the people will join on another peer and then they'll just download as many blocks as they can before they run out of storage and that'll essentially dictate the proportion of the time that they can hash on new blocks in the system thinking about other projects that are tackling this I feel like what I've usually understood it as is more like you have a blockchain and then you have a place where data is stored and then there's sort of check-in points but what you're describing is something else yeah it's completely different okay so the data itself actually is the blockchain yeah I mean one way of looking at it is saying it's like untain data storage that scales so what we're looking at is permanence of data storage most of these other in fact I think all of these other storage projects are we

just looking at decentralized storage we're looking at slightly smaller quantities of data when making sure that data is around forever so if you wanted to I'm just picturing if you wanted to like have this blockchain if you want to view at a client your miner you're holding a ton of data like this is gonna be a huge flat chain yeah eventually I mean we we hope for it to get extremely large that's kind of the point but the idea is that you know if your miner and you can only store a terabyte then you just do a terabyte but say it's 10 terabytes then you can only take part in a hashing computation ten percent of the time I see so you basically it's you wouldn't be punished in the way that some other block chains would punish you for not having the full thing you're expected to

not have the full thing precisely so we have a sort of competition based approach to storage rather than a contractual based approach so most these other projects they go off and they say hey I want six people to store replication of this data for this period of time and I'll pay this much money for it and then the blockchain is essentially used to like settle payments for that we take a totally different approach but we say no you should just compete to make as many copies of this data as possible and the net effect of this in practice because we've been running since June of last year is that if you add a piece of data to our weave today you get a hundred and fifty replications of that piece of data which is um yeah it's like the highest level of data replication available in any

commercial system right now I want to dig into a bit more on the on the guarantees of that system but I had two questions that just popped up one is you say you choose this old block in some pseudo random fashion where does this randomness come from yeah it's actually really simple we just used the independent hash of the last block which is not the dependent hash which is sort of what you used to feel from the kind of a Bitcoin world say it's a derivative of this but this is fine because the only thing that you can do by manipulating that value is make it so that you have access to the next block that'll be required but of course you had access to the current block so you have to throw away candidate locks that a valid that you would get a reward for

just in order for you to take part in the hashing get a potential reward for the next block so yeah you can technically kind of choose which block but there's no incentive to do it in fact actually the yeah it's a big Oh n square or difficulty squared problem so like there's really no incentive but yeah it means that we can keep the protocol super simple but I still am not clear on exactly where does the random where does this random so it's an output of well actually it sort of 2-step derivative of the proof of work of the previous block you have a pseudo-random number that gave you some other rent like you've taken the hash of a random block into your proof of proof of your work thing and then you use that to then generate your next number yet what

happens in the system is because it's a block we've not a block in order to verify an old block if we didn't have this thing that we call the independent hash you would have to have access to the recall block for that old block and that's of course a recursive problem so you would end up if you want to verify I don't know block 10,000 the complexity is like I think it's a Big O in fact yeah no no no yeah Big O log - very good so we have to have this system that we call an independent hash which is like once you produce the proof of work so you take the data segment that's including all of the contents from the record book you find a nonce that satisfies the difficulty requirement you then take the

transaction IDs from the current block and all of these sort of metadata and then you hash that again including the dependent hash and now you've got what we call the independent hash and that is the randomness number the open days you can use this to derive random okay yeah you also mentioned that hashing was more expensive than storing the data which hash like which proof-of-work algorithm are using which hash function yeah so at the moment I mean underneath the proof of access layer there's sort of normal proof of work underneath and it's just char - - sorry char - 384 at the moment but we've got a gen prog implementation that we built that were kind of testing out we've been running a testing out of that I think since early December but we're just sort of like leaving it going

everyone seems very tenuous about the idea of gem prog like it's really nice in principle and and the the logic behind it is excellent but like nobody's really done it in practice at scale so we're kind of waiting to see how that goes so when I was looking through the light paper what I understood was that you have these miners that are grabbing kind of random blocks creating the new block block not blockchain block weave but then you also have this data structure is that the thing that's sort of binding this all together it's block weave data structure yeah exactly so when we try to explain to people it's kind of interesting like it proof of X s is a an emergent property of the block we've data structure you

can't really have a block leaf without having proof of access so they're very closely linked and yeah that certainly ties together the sort of storage part but then the the delivery of data part is is basically off chain I'm different speaking of delivery something that all of storage projects have to deal with is how do you actually use the data that's stored and how do like a normal person start up a node and get access to the data and and sort of what's are there incentives around this are there like is bandwidth usage incentivized etc like how does that work in our way yeah absolutely it's essentially the same competition like approach so what we've done is taken something very similar to

optimistic tit-for-tat which was what BitTorrent used way back in like 2007 2005 or so even and still using right right well it's kind of modified nowadays so they've got like better systems but basically we've taken that is the the core principle which is like if you give me some data I'm gonna give you some data nordicist sort of kickstart that system so that it tends towards everyone giving everyone data then there's a little bit of optimism at the beginnings like I'll try giving you some data and see if you give me some data back and when you do then then the torrent or swarm or whatever you want to call it starts to flow yes we use a system like this and on top of that there's a data distribution incentive relating to transactions and new blocks so obviously is a miner you want to get

access to transactions as soon as possible as well as new blocks like this is critical to being an effective miner so what we do is we say okay well if you serve my data requests not just quickly but also like reliably you don't say you've got a piece of data and then just flake out then I'm going to prioritize you for the distribution of those new transactions and blocks which makes you more efficient miner yeah so you have this sort of two layer incentive to the data distribution mechanism but you don't have any concept of like a light client in there where you know I am a user who I don't really want to store any data I just want you know to get access to this website that I'm trying to get access to or whatever and you know I want this small piece of data I

don't want to participate in the network as a whole how does that work yeah I mean essentially the nodes are trying to trick each other into thinking that browsers to begin with so when you an attach a browser to the system it just looks like enough node in the network yeah this is a kind of interesting area because there's lots of things about that that a you'd think it would be a protocol level incentivization system and it is to some extent like it's a pretty basic concept but the intricacies of how you get that to work they're all about the web stack which is really messy in context topic but there is so yeah like going back to like like lines structures in theory in for instance a like client can get a piece of data and like a piece of witness data was like in the Merkel tree

and prove that yes this is the right data you're giving and and like this is all good and I can proceed and full nodes who serve these like clients they in a theorem they have no incentive to do that and there is like a credit based system they just give our credits for free currently and it mostly works because of altruistic behavior but I mean there's lots of discussions on how to incentivize this but it's it's extremely hard like you'd have to do it on a per request basis or whatever have you guys thought and at all but like how you incent ice and like do you have this proof structure as well like you can prove that you're giving the correct thing yeah yeah this is mostly about gateways actually so so basically in the incentive structure we'll just sort of

piggybacking on the way that the nodes talk to each other to begin with but about proofs you you essentially trust a node in the network or you run the web extension that we built which is essentially kind of like a lie client and then you can download the data metadata that you need in order to prove that something was part of the block we've put a certain height then you take the transaction you prove that but we think that actually it's going to be really important with these sort of web on a blockchain or web in a decentralized storage network systems that your user should not have to be able to should not have to do anything in order to be able to access the data it should just seemed like the accessing a normal website and maybe they lose some level of trust lessness when they

do that so in our system you would connect to a trusted gateway and that gateway would serve you the data as well or rather they would serve your frame which allows you to get the data from somewhere else in the network and then verify that the data that you've got is actually legitimate but yeah it should be a kind of see-through process the user should just think that there accessing a webpage which is actually what we've achieved at the moment pretty neat it's funny just before we were having a conversation about the word trust list and trusted I'm kind of curious what you mean there so you would access a trusted party in order to get into the trustless system yes that's exactly what is the trusted party yes so you have to choose a gateway to talk to this is that a minor - can run gateways

but they don't have to and gateways are specific things in the block we like where does that live what's your gateway yeah it's actually just a small package of HTML and JavaScript that you need to serve to a browser that will then allow that browser to go talk to another node in the network so it's got a bunch of peers that the node that you spoke to in the beginning the Gateway told you about but then it allows you to verify that the other things in the network that the people are giving you are actually correct I mean I would think about it that's a sort of ipfs gateway I mean they do the same thing right here you're not in it like I mean conceptually like you go to ipfs dot io slash whatever hash ipfs dot io can be owned by any random person they can give you any

random data there's no guarantee that they're giving you this thing you are trusting whoever owns ipfs that io to serve you that gateway correctly but you can set up your own gateway to if you want and serve yourself through that yeah it's kind of like that but slightly different so when you connect to ipfs dot IO or the CloudFlare run gateway then they're going to serve you the data that is held locally what we do is give you a small frame that you're it's essentially a program that your browser can run that allows you to download that data from other random people in the our weave Network and then verify that the data that you're getting back is actually legitimate well which really cuts down on gateway traffic so it gives people more of an incentive to run the Gateway in the first place which I guess

like in an interim world would be akin to the Gateway giving you all the headers and then you go out and like request data from the rest of the network and you can validate that data against the headers that you're given yeah that kind of thing yeah but you don't actually have to use the Gateway so if you install the our weak web extension then if you go to an R we've done that stroke something URL then it takes you to a node that you already know in the network and of course you already have this sort of program that fyz the days of your getting yeah inside the web extension so you just do it that way no gateway totally decentralized how was the weather extension built like is it all JavaScript or like are you running native binary something that you're downloading yeah it's just typescript

using a subtle crypto library which built into the browser's nowadays although honestly crypto libraries in JavaScript an absolute nightmare yeah it's really unfortunate that we're in 2019 left in this situation where you know you're trying to convert between pens and JW KS and it's just you know we really shouldn't be having to do any of this anymore but yet here we are are you gonna be writing the libraries we've thought about it really carefully but then we also figured that what you end up with is is all of these sort of semi complete crypto libraries that do this and that and that thing but not that and that and that other thing yet that we essentially have clogging up the ecosystem at the moment it would be much better if there was a sort of serious

standardization effort that made serious the well-vetted yeah crypto infrastructure for javascript programs but you know we live in hope I guess I actually just open up a browser I'm looking at a website that's hosted on our we've mi all the sudden a minor or no no no no comma minor so you go to our website you click on the become a minor button and then yeah you download a client locally on Linux or Mac machine yeah and you run basically what is essentially an Erlang program that just represents the minor and then I pick a percentage like do I pick a percentage of the block weave that I'm going to a host no you pick an amount of data you

plan to host um yeah if it was a percentage of the block weave then as it expands you found it would get bigger makes sense so I'm like okay I can host this much so a lot of different people or individuals will come and become miners through this process and they say ok I'm gonna host this much data what if you have what if the block weave is so big and there's not enough people hosting enough right this is the core question so the answer to that is that we have indicators that tell us basically when when data storage is becoming guess in that work and as soon as we get like to a position whether less than a hundred replications of a normal piece of data then the price of storing a piece of data increases dramatically and

when this happens either one of two things happens right either the people storing data stops throwing so much data or they continue to store the data but the reward for taking part is a minor increases higher reward for miners mole miners join the network more storage capacity price goes down again I want to dig into that because like I know that you have a pretty unique miner or award structure so like how does that miner award work like what's that it's not just I mined this block I get one Bitcoin kind of reward right yeah exactly so what we do is we have a carry over pool in each block and when you add transactions to the network the rewards

go into that pool and then you take a proportion of the pool relative to the size of the recall block that was required in order to mine that block so basically you plot a kind of sigmoid curve where the the 50% point is on the average size of the block in the network and then if it's much larger than that then you take a larger take from the pool of this much more you take a lesser amount so this essentially means that you're providing an ongoing incentive for storing any piece of data through what's kind of like a a storage specific smart contract if you like really it's just you're making a contract with the network that says hey can you make sure that you give out a small reward for the storage of this piece of data indefinitely and obviously they will all get smaller over time and this is part

of the consensus and gets validated when you yank a block so someone is trying to take a larger portion of the pool then you know that's an invasion they've worked off the network so going back to this question which i think is central to like a lot of what we're talking about is what are the guarantees of this system like do you have a guarantee that your data is available forever do you have a guarantee that it is stored forever even if it's not available and like what what are the guarantees provided and just knowing what I know I I feel like it has to be probabilistic guarantees yeah so there's a guarantee that there's always an incentive inside the system to

produce the data on demand and to perpetuate the data going forward the number of replications is probabilistic but it's sort of set such that there's such a large what would you say like buffer zone that there is always going to be a very large number of replications and what what determines so you started talking about this what determines the probability of my data being stored forever like what are the parameters that go into this because like you say like we came to have a hundred replications and then prices go up if it's less or like what are the parameters that go into that well I guess there's all sorts of things I mean its price of token of you see would be an important one number of tokens in the

pool yeah I mean that there's tens hundreds of factors but the goal is with extremely high probability your data is stored forever and available for retrieval at any time this is the point yet like the whole system is engineered such as that incentive overrides pretty much everything else are there cases where people's data could actually be lost well there's probabilistic so theoretically it's possible but the likelihood is extraordinarily low I mean if you if you have data replicated on 150 miners if all of those 150 miners burnt their hard drives then they would obviously lose data I'm you're more curious though like something that's

more likely to happen is there's a bug in the software of the miner and accidentally delete something that's an interesting one yeah we we don't delete anything inside the miner so we just move data to a sort of retired directory yeah we're we're extremely careful to make sure that this kind of thing yeah can't happen or at least yeah it hasn't exceptionally low likelihood of happening of course if you have an internet-connected computer just running our weave or any other kind of piece of software there's a very low chance that someone will find some kind of yeah remote code exploitation bug but actually we're running the system on top of Erlang which is extremely well tested code this is the telephony frame

essentially that runs like you know 20% of every major Western countries yet telephone lines so if there were remote code exploitation systems or problems inside Erlang we would probably found out about it like 30 or 40 years ago what if okay so different maybe scenario but what if like so 150 miners are hosting all this data the data pool the the full thing becomes very very large and then those miners stop participating what would like I understood that the price would go up and it would incentivize but what if they still dropped off then it doesn't work well I mean there's always a potential that all of the miners could leave the Bitcoin

network and then you know or there's a remote code exploitation problem in Bitcoin and it just deletes all of the copies of the the blockchain in in existence but actually one of the interesting things we've seen with the network so far is that it's kind of like a a meme in the door Keynesian sense that produces more and more copies of the data regardless of whether those dates or copies are actually connected to the network so what we find by talking to miners that have turned off their machine for whatever reason is that they haven't deleted the copies of the network like they've just left them on their hard drives and over time those old copies accumulate and what's cool about the network is essentially if there's a piece of data that very few people are replicating or nobody's replicating that's connected to the

network it'll basically just wait there and the storage are the incentive for someone to go out and find that piece of data continues to grow so if you have a miner that you turned off and then you sign up for like a mailing list that says hey we're looking for this block basically if the if the chain just stops for a while you say oh that's interesting I would get a very large reward if I turn back on my miner and sure enough you reconnected to the network and then you reseed that data to everyone else and now that copy is back in existence so and another thing to cover is that the the incentive for storing a piece of data increases as the number of people storing that data decreases so this is essentially because rare right the reality is the more valuable it is so you incentivize to

seek out those rare blocks and make replications of them because essentially if you're the only person in the network that has a copy of the data if that data is chosen as a recall block then you're guaranteed the reward if you're online whereas obviously normally you're competing against hundreds or thousands of other miners is there a way that I can censor the distribution like can I create a block and make sure that I'm the only one that holds a copy of it well I mean if you don't distribute it to the rest of the network then I guess in principle yeah like it blocked withholding attack like you will any other block team but you wouldn't get any reward yeah but you wouldn't get a reward and the data wouldn't be mirrored by anyone else so it's kind of like you've made your own fork I guess you're

sitting there being like it is mine yeah but if you could distribute it like get it included but make sure that you're the only miner I've had it and mine right got it no so you can't do that everyone else in the network needs to have a copy of the data locally to verify it and add it to the block weave so yeah there's this sort of leveling the playing field that happens in order for you to get the reward for the block everyone else has to have it at some point they can then delete it if they want to but like yeah they have to have seen the data this might sound super random but like where does the miner reward come from in this system is it just generated like it's you guys have a token mm-hmm miners just get some of this token yeah yeah

yeah so the rewards are in the token but they get some of the reward from you know sort of typical inflation schedule and they get the rest of the reward from the TX reward pool which is that one that's carried between blocks and a small portion for literally mining the block that those transactions going to so I guess it's three sources is there like if someone's trying to access the data they are paying something no we bundling the price of access into the initial payment to put data on the network yeah exactly so when you want to insert the data into it that's when you pay yeah and that payment is then distributed out to the miners exactly yeah plus other things plus inflation rewards but overtime okay

what do you think of the storage fee discussions and most blockchains now are having of saying sir we can't let the blockchain grow indefinitely we have to put up like paying ones for infinite storage there's an incentive misalignment and like how do we yeah keep things around forever yeah it certainly doesn't work if you just have a system like Bitcoin or aetherium where you pay once you get the transaction put into a block and then all of that reward goes directly to the miner and that's just it that's the end of it no that that doesn't work with our system we make the storage of that oh look part of the consensus mechanism so essentially what you're doing is you're offsetting

value that would otherwise we spend on hashing on the storage of data that's our approach oh and well that's one layer to the approach the other layer is you simply work out what the perpetual cost of storages and then you make sure that the TX reward pool is yet distributed over time such that it mirrors that and actually like the perpetual cost of just storage on its own is fairly simple to work out like at least at this point so you can look at technologies they're kind of fifty maybe a hundred years with us pushing it down the pipeline and you can look at the theoretical maximums for the data density data access and you can see that we're like vastly vastly far away from those limits and you can also plot backwards in time and you can say okay

well storage cost decreased along this sort of curve we can see that out here there are technologies that are going to sort of follow that curve as well yeah and then you can essentially just work out what the perpetual cost of that storage is it's a really interesting topic actually we talk about this all the time we have a PhD economist in house lidar yeah this is kind of its favorite topic yeah I mean I think addressing it through you kind of have storage fees by just the incentive model but I guess it's inflation driven like the person that pays for storage in the first place doesn't keep paying to have it stored they do that their tokens go into the reward pool which is then slowly eked out over time in decreasing

quantities so the idea is that as well as the competition based reward where a sent to the offsetting that value that's otherwise spent on hashing on storage you're also making sure there's a profitable reward in tokens from the token reward pool yeah they're just compensates - for the storage of that old data right and this is the bit that's just like a kind of um it's it's almost like you're setting up a contract through the network to just give out these tokens indefinitely and the nice thing is the network you know has very low overhead of cost of running so you can do that very simply so out of the pool there's a pool and you're going to be continuously paying out these fees to miners there is inflation on that

pool no inflation is elsewhere your walk rewards okay and sort of you sort of mentioned it here but the the payments out would be smaller and smaller smaller as time goes on the assumption then being that the price will go up no no no no the the payment for old data because decreases over time in line with the expected increase in storage dentists density and the decrease in storage cost but that's not the primary mechanism that sort of backup mechanism if you like the main one is just that we're offsetting the competition with hashing that goes into you know electricity wasted basically with proof-of-work systems with just

storage of data is there a cost to serving it to serving the data so that's competition based as well so essentially what you're doing when you pay to put a piece of data into the system is to have that data added to the consensus mechanism itself and part of that is also the incentive mechanism for moving other bits of data around the system I mean more like if it's try it so when it's accessed is there cost not at the point of access no okay but it just like in life is there cost is there a when I go check out something and pull it out is there a cost of electricity is there cost somewhere along that line that could be incurred by the miners so there's definitely a cost to miners of storing data and said sure okay

absolutely but we've essentially bundled that price into the initial price and then yet the miner is getting paid for that through the tokens reward the reason I'm asking here is like what if one piece of data is super super product I was arrested is like not so what happened so all data distribution mechanisms follow a sort of Pareto distribution there's very small pieces of data a very small number of pieces of data did a very highly sought after and then there's a massive long tail of yeah data that people want once or twice in the whole of its storage length yeah so what you do is you just a more ties out the cost of that data serving across all of the data on the topic of storage fees

I just wanted to bring something up and and I wonder if you've looked at this and and this is sort of the problem in the theory important since I keep bringing this up because it's my main point of reference but the problem there is not the cost of storage like the the state size is actually not that large it's like 12 gigs anyone can have that on their computer is not a problem the problem with storage is the performance of this system as a whole degrades the larger the storage state is so twelve gigs of data is not a problem to store in like drive space but 280 million tree nodes is hard to manage in a database and the performance of the

system degrades as the like as a sort of secondary effect of storing more data is this something that you've analyzed in your system and like what what's the performance aspect of the system as it grows yeah so um confirming new blocks is constant time so bigger one essentially so more data added to the system doesn't add to the amount of time it takes to verify a new block and the way that we do this is essentially we just don't entangle the block of the data stored in one block with the data stored in another block in the same way that sort of traditional tree based systems do so yeah of course if the theory or more Bitcoin you end up with this sort of ever-increasing tree yeah we just don't have that because we

split the blocks differently and this is coming back to the independent hash system this is so that we can do that we need to be able to take a block from someone else in the network like at the time of verifying a new block and have them serve that to us and then say oh yeah ok sure that is actually the old block I don't need to um yeah I don't need to sort verify the the Ricoh block associated about etc so yeah it's it's constant time but that was something that um we actually changed in a recent hard folk because the original version it would a slightly increased in complexity for each new book but now it doesn't it's constant what exactly do you see our we've as replacing what place does it actually take yet so we don't actually

see it as replacing anything we don't think the permanent information storage in this way has ever existed before like the real goal if you want to think about it in that sense is to remove the existence of the idea of a memory hole that is a place where you can put information that was once public and well known and such that it can be forgotten to all humanity and stopped affecting our decisions made in the future and that was what got us into this in the first place we were really interested in how you store actually with the potential of storing news articles on a blockchain that was what God has started in the first place because back in not 2016 there there was a sort of rise in the phenomena of fake news and one of the one of the tactics

that happens in information warfare is not just the spreading of fake news but also the retraction of that news later it silently without any turning it down yeah yeah and that's a real problem because of course these pieces of information affect the way that people make take decisions and so in Ukraine you can see very clearly I won't go into the geopolitics too much but you can see very clearly that some of the sides in the operation were attempting to affect the way that people acted with this fake news and then removing the sources such that no one knew where that data actually came from so we weren't fans of this we thought okay well blockchains a kind of good data integrity and making large copies of data and so we started with that and then you know two years later we ended up with the block reef

and actually recently we so talking about Ukraine again there was the Kerch Strait incident where some russian naval ships let's not say attacked but engaged in some fashion some ukrainian ships at the the curve straight of the sea of avaaz and there were news articles written about this by russian propaganda outlets one particular one by Sputnik and people from our community he managed to make a copy of that data on the our we've fine at the time that was kind of interesting this history on a blockchain that's pretty cool so we wrote a blog about it but then a couple months later we realized that actually that original sputnik article had been deleted the Russian government had attempted to censor their own propaganda on the issue but they couldn't because

it's embedded in the block or even there's 150 copies distributed globally and they simply can't censor that information now but there is I mean there's like the what's it called the archive web yeah Internet Archive yeah the Internet Archive we make fans of the Internet Archive but it's centrally managed and controlled and also you know they're just as susceptible to fires as every other archive like there was in fact a fire in 2016 where they lost some data that had just been digitized I think so they just about avoided having sort of catastrophic data loss but this is a pretty serious issue if you have a centralized archive you are all his it's susceptible to hardware and I guess disaster type failures you know the fire

flood all of these things so we think that decentralized archives are really the way to go the Internet Archive is also mostly paid out of the pockets of a few rich people so it's yeah it's literally not sustainable yeah and and also you know that they're a single person to go to and ask for data to be removed which does happen whereas in our system we have this thing that we call the the decentralized content policy so each miner in the network can choose which data they want to store or not to store and also they vote with their hashing power on whether pieces of data should be allowed into the network or not as a consequence of this at the sort of meta level you get group consensus about what sort of data

should be in this archive very so what's this part so this is like a it's like a governance part or something you're voting to accept it in yeah I mean it's actually really simple at the protocol level all that's happening is you have sort of blackness of content that you won't accept so it's the moment is hash based or substring based so you can block content based on whether it's a piece of known data you don't want or whether it contains some piece of data that's sort of repeated that you don't want to have access to and when some since you want those transactions you'll just delete it and so if somebody sends you a block later which contains one of those transactions you just run accept the block and so this happens at a network level it causes essentially a vote using

hashing power see whether the data gets into the network or not but can you actually are the miners actually able to say we don't want a certain kind of data like a actual website so on hash created but like like you know that stuff yeah of course you can do substring matches on website names and that would block it for now but one of the things we're working on in the future is making it so that people can call out to external api's and essentially delegate that task of black lists to someone else so google offer an api they'll take any image and it will find out for you approximately what they think is inside that image and then if you're minor you can just decide based on that yeah whether you want to accept it or not

similarly there are sort of other api is for illicit content detection that kind of thing so our idea is to make it so that the miners have this sum they can use the tools that the big centralized web powers now use to protect their systems from this kind of illicit content on their own machines that's where we're heading with this can anything be deleted not deleted so well when you write a piece of data into the block we've what you're essentially doing is paying to have it added to the consensus mechanism this doesn't guarantee you as we spoke about that there will be a copy of it just makes an extremely high likelihood that there will be so we essentially have a system whereby content is voted in on a consensus basis so 51% or more say it

should be an and it gets into the network but it's voted out on a unanimous basis that is you have to have every single miner in the world decide not to store the piece of content for the content not to be there this kind of goes back to that question of like could it be lost but here it's like could it be banned yeah yeah exactly I mean it's we think that the the really interesting thing about decentralized networks is the competition based approach to typical economic activities so no longer do you have to make a contract with someone that says I would like you to do this for this length of time and I will pay you this much which by the way you know there's no guarantee that everyone will abide by their contracts this is of course a problem

that you get in and storage and seer and so on yeah we take the inverse approaches we say okay well let's make sure that everyone is competing to make or to do as good a job as is possible to fulfill this thing that you've paid them to do I feel like that's a really interesting aspect that I haven't really thought about like how do you do content policies and I mean there I assume you run into a bunch of intricacies around like you said you have some way of measuring like the distribution of this piece of data is roughly around this number of miners but then you introduce you know now it's not just among miners it's among miners that don't have a

content policy that excludes this content like you you're like who the data is distributed among has quite a lot of factors yeah yeah it's only a fairly complex distribution system that's for sure I thing incentives apply you know if a piece of content is mirrored by a small number of people there's a high incentive to store that content how do you wouldn't agin to this before but how do you detect how many people do store or something yeah this is interesting so we don't detect it directly because there are questions of basically people faking which we have a mechanism for getting around but we didn't make it a core part of the

protocol instead what we do actually is look at the hashing rate in the network and use this as a an indicator for the likely number of replications of any piece of information so if the effective hashing rate is 10 Peter hashes say and then we're only actually seeing 1 Peter hash then we can say that there's approximately 10 percent replication of the content in the network something like that but getting back to this question of what we see of replacing we we don't really see it replacing anything we think the permanent information storage is a new kind of thing and we also think it offers something that people simply cannot buy in the centralize world like there's something particular about this case of decentralized replication of data that you just can't get if you go to Amazon

you say hey replicate my data table okay sure I can do it in these and these in these regions but it is just not the same level of integrity as you get with a blockchain so yeah we don't really see that we're competing against anyone in that sense which is quite kind of exciting like part of the interesting job of adoption is to sort of educate people on why permanent information storage might be important the only thing I can compare it to offhand is the story that Zuko tells about how there was like a love letter in the blockchain in the Z cash blockchain what was it it was the data I was like the data was in the data field there's like a hash of a document that was a love life yeah I mean a lot of ball chains have stored

data it's just storing one gig on Bitcoin it's extremely expensive yeah yeah I don't know if you've actually said the word perma web yet but I know that on the website you have that listed as one of the things that you guys are building or building towards sort of a philosophical question but I I was trying to think if like a permanent web is a good thing yeah so there are obviously pieces of data in society that should not be permanent like we don't apply this isn't as a generic storage solution because it's permanent there are certain types of information that should be eternal and many other types of information there should be ephemeral so things like we're working with some

stable coins to store their attestation documents let's say how much is in the bank account which backs the stable coin this is the kind of thing you want permanent record of you don't want to lose those piece of information but you know like your private am messages with your spouse that's obviously the kind of data that should be ephemeral so we make no you know bones or claims about this being generally applicable to every single thing it simply isn't and but actually one of the things that we see that the other decentralized storage systems might have a problem with is that you can buy storage that's fairly well replicated you know enough if it's not that important to you very easily and cheaply from Amazon or Google and

decentralization where obviously I think I assume we're all in agreement is a better thing general but it's very hard to get people to make a shift like that particularly for their critical infrastructure so yeah I mean this is up to them to show but we see that they might have a hard time just selling it on the fact that like yes okay it's storage but it's decentralized this time we think you have to have an extra angle and that's what we offer with permanence so I guess what you're doing is you're focusing men on this archive the archival websites the things that would need to be kept for a long period of time and people potentially would pay to have that kept for a long period of time and not for

just general websites like my blog I might not want it to be permanent I don't know maybe I do I'm like examples you're right example I think that's actually like a personal philosophical question I think there's a lot of people that do want their blogs to be permanent so one of the things that we can offer is they're like okay you can add your website to the perma web say it's a blog and then what you're essentially doing is is adding your thoughts to this sort of massive collection of human history and knowledge which will be perpetuated for generations to come so in the same way that we look back at an Anne Frank's diary so these kinds of things from World War two this is a generic system for sending those messages to the future in fact one of the ways that we've been

sort of conceptualizing this is that the internet connected people across large areas of space extremely efficiently what we would like to do is connect people over very long periods of time in the same way and that's why we've kind of built a web on top of this thing so briefly about the perma web I mean it's decentralized storage network so you can load and serve HTML and web sites across it to a browser and as a consequence you sort of get a web on top for free so you can make a website and one of the things we've been looking at is just simple boring or not not boring but simple kind of easy developer hackathon type things that you're putting together like I've got this

little JavaScript application that does this this is kind of cool but I don't want to pay Amazon like seven dollars a month to run a server to keep it online I'll just put it on this thing and it cost me into a fraction of a penny and it's there forever it's the kind of thing that I don't particularly care if it's forever but I'm not going to be upset if like it's still there in 20 years time right it's just I need somewhere to store it simply and cheaply you sort of touched on some other projects and what are your thoughts on the other projects in the space in the decentralized storage space I would also be curious to hear yeah like we talked in the previous episodes about sort of this p2p winter and like how the the improved UX of centralization took over and then how

it's now kind of coming back like what's your thoughts on this state of decentralized storage huh that's an interesting one I mean the first thing I guess is the way massive fans of BitTorrent as a protocol we think BitTorrent is the coolest thing the other incentivized storage projects like I said before I think they're gonna have a bit of a hard time gaining adoption because it's not immediately clear to people outside of the space what the advantages over just centralized storage yeah I think there's quite a way to go on the adoption side there how much does it just out of curiosity how much does it cost to store data on are we right now

one factor is that the price of storage of a byte increases for every added byte this is to incentivize people to keep transaction sizes small so if you have a very large file you split it over many many transactions which means we can sort of fit it into blocks in a more efficient manner then we save you money on your storage but if you really insist on store egan and one big thing and will allow you to do it but it's just more expensive but say for I know we just before I came on here we were hacking together a kind of decentralized website decentralized mail system on top of this web so sort of webmail but it's all decentralized it's secure permanent and I think to send a message it was costing something like a hundredth of a cent

that kind of thing we're talking in the range of a cent per megabytes ish approximately yeah another thought I had while we were talking was curious like not are we related really but a curious speculation is you could like you said you could delete data from this network you requires all the miners and everyone in the in the pool to agree that this is something we're gonna delete and you go and do that you could have like completely off chain coordination where like someone goes around to all the miners saying hey I'll pay you a hundred bucks to delete this thing but you could also do that in protocol you could say like here's a transaction you know I prove that I was the one that

put it on the chain I now want it off the chain and I'll pay all of you this much yeah you would yeah that's an interesting one we wondered about having a sort of burn list the the the same user to put the data into the network can come back and then yeah ask for that data to be removed or no longer mirrored so for a start they can come along and they can say hey like I would like you to delete my data and there's a special kind of transaction type that you can issue and then it's up to - to decide whether they honor it or not the yeah we've thought about the literal concept of just cancelling the the essential permanent contract for storage and we think that it sort of breaks the core use case of the system like if you can

be coerced to delete a piece of data then it's no longer a permanent record of human history like there are many places in really what we were attempting to stop is the idea of book burning in the digital age and there are many times in history where people have been you know coerced to a much higher degree to remove information than would be required to just get them to issue this retraction transaction so it's not currently on our agenda to add it to the protocol but you know like it this is a seriously decentralized project we don't own any of it so if the miners wake up tomorrow and I'm like hey guys let's make a system for revoking the contracts then they can definitely do that but it keeps reminding me of is like the library in Alexandria and they're burning down yeah and have so many

copies of so many interesting pieces of work were ruined and now there's only like echoes of them like text referencing text that were gone right I mean that's it just keeps appearing like they're completely Sochi reminding me of that what's particularly funny about that story is that they actually lost all of the records of losing all of the records at the Library of Alexandria so we don't even know how it burned down or even if it actually existed its yeah at this point is completely lost in the mystery though in the like yeah I like that I mean just how many manuscripts and everything were lost and then the only way that some of them survived was because there had been copies somewhere and other exactly like replicas is almost decentralized replicas in fact one other point I want to bring up Frederick's favorite topic

gdpr that sarcasm there he's cringing right now um I I wonder I mean when you're talking about having something deleted and having the miners choose it's really it's an it's a topic that constantly comes up when we talk about like zero knowledge stuff and we talk about blockchain stuff so how are you thinking about CDP are it's pretty simple from our case in fact is actually better than other blockchain systems from the point of view of miners because if somebody says hey don't don't store my piece of data then we have a mechanism that allows them to stay a member of the network and also delete that information it's that simple from our point of view but what if the miners say no well the miners are in gdpr parlance data

controllers it is their responsibility legally to deal with this issue we give them all of the tools that we can to make it easy but they are as is sort of a point of decentralized networks free agents you know they're responsible for themselves and also they can earn money for themselves so as a consequence they must decide yeah essentially which request to comply with I mean it's almost brings it though like so that sort of makes them also sound like censors like they could yes it's so funny because there's almost like this line that these miners will inevitably have to walk which is so for example say something is saved that currently is deemed fine and in 20 years is deemed

not okay content they could take it down and then they would be acting as census yeah arbiter of what's allowed to be of course permanent yeah we don't try and escape the fact that no the miners are senses there are definitely sensors I mean it's sort of we actually come at it from a different point of view and we say okay well in the decentralized network you you can't force people to do anything so it's sensible that we give the miners the ability to decide what they want to take part in and of course if no one wants to take part in an activity then the activity doesn't happen but a consequence of this is absolutely they're acting as sensors there's no getting around that

but it would be did you say it was a hundred percent of miners would need to agree in order to take something out yeah I mean what you're really just asking is for the data not to be replicated dot and so you have to get everyone in the network to see my debt not to replicate the data it's hard and it's also I mean beyond that to some degree because the miners kind of Canada can reject it from the consensus and from replicating it further but you like as long as any full node has joined this network and stored everything you knew even if they're not a miner they still have that data you can't force them to delete it as well like so it depends on how far spread out everything is this this is like the web right and they're sort of I think one of the things that

in our society we're going to have to come to terms with over the next 10 20 years is that as soon as the bits leave your laptop or your computer you don't really own them and you can't be sure that replications of those bits don't exist somewhere else and that's just a fundamental effect of the the networks that we started building when when the internet came about and it's not going to be something we can get away from in fact really as soon as something is in the Internet it's in kind of a quantum state where it may exist permanently but it also may not there's no real way of knowing you mentioned adoption a little bit earlier and that's something that's interesting to me and so we've seen how block chains try to gain and

option with developers or with users or just like meta masks which is like an adoption project but it's kind of centralizing things for like moving things in different directions how do you guys think about adoption and like particularly being like you're not Bitcoin or ethereum so then you kind of have to deal with the reality that people will hold multiple tokens and if it for instance one of the use cases is you know I I store my entire front end I saw all my date on our we've and I store my application logic on aetherium now I need two tokens and how do I deal with that yeah I'm curious to hear our thoughts on on this

sort of space yeah absolutely so in terms of adoption I think um it reminds me of this section from Peter Thiel's book zero to one where he's talking about if you really want to get someone to adopt a new piece of technology it has to not just be better than what they're currently using it has to be 10x better in some set of weights you know more than one way and we really think that that where we shine is we are at least ten times better than any existing archiving system in the world this is pretty clear to us and we're attempting to make it clear to people that we speak to and yeah that's actually going really well so we see that lots of project well there's kind of two sides to this there's the decentralized space site where people just need incentivized

permanent storage to go along with their blockchain appliance of some kind we can offer that very very simple but outside of that space there's a whole bunch of places in society where you need to store records for a very long period and there's no real system that's set up to do that at the moment so we're actually seeing a lot of advancement at that site for example we're working with a group of architects to make a system that stores architectural drawings permanently the idea being that because for regulatory reasons they need to store the copies of those files for 40 years anyway and can be finding a massive quantities money if they don't yeah this is something that we can very easily fix for them and that they can't get from anyone else in society if you go out and you say to Amazon hey I want

to store this piece of data and I yeah sure fine just give it to us we'll store it what's the problem the problem is that they're definitely not going to be offering that same service in 40 years time likelihood of that is just you know tends towards zero so from that point of view we offer something that's pretty unique and that's very helpful for gaining adoption regarding the multi token system yeah we're quite sure that they'll be one maybe two utility tokens that people uses the sort of if you like mother token and then they just buy quantities of the other tokens they need on demand it's not clear what that sort of meta token will be yet but you know and well could plausibly be Bitcoin with you know lightning networks on the side to

settle the payments faster something like that and but that all remains to be seen but as we understand it there will be like people will buy the token on demand to use the storage is our expectation eventually your architecture example is that actually gonna be used on the are we've block we've or is it it kind of private no that one will be public yeah oh it is public it's on the main one it will be this is this is not yet pushing on chain transactions but many of the others are do you see projects asking you if they can do their own kind of like private version times could use a fork what you've done and

then make their own little yeah I mean you can circle do but the problem is like it only works because of the incentive net mechanism which gives you safety in numbers so you can make your own private one that that works pretty well and we can also make systems that say okay you got a private one but you submit the hashes from the transactions and the hashes from the blocks to the main one and then you get some kind of security guarantees but you don't get the safety in numbers you get with the public are we've and that's pretty critical to the way the whole thing works in fact I think private blockchains in general I'm not a particular fan of I think they don't offer nearly as many benefits as public blockchains you know fundamentally if you're running a private blockchain before I don't know into business

transactions or something like this something we're a very small number of people are involved if it's the kind of thing where I can just boot up another ten nodes and very quickly rewrite the history of the thing then you're not really offering that much above a database and someone that just loves decentralized tech I think this is pretty boring honestly yeah I mean I think in your case they're really like you couldn't really run a private version of it because you don't have civil resistance in like how this thing works and the entire points from this start is you know you have other people storing things for you and you pay them like if you are storing yourself you're just paying yourself like there's no it's a real incentive mechanism around

it yeah whereas with like you if you're running a private aetherium network usually running it under a different consensus algorithm then then maintenance so what you're really just looking for is a Byzantine fault tolerant database that also has contracts like that highest distributable code yeah I mean it's sort of like it's a blockchain nominally but it's they're not after the decentralization aspect of a blockchain they're more after the operational efficiency and like there exists other Byzantine fault tolerant databases but they're really hard to set up for they don't have these features or whatever yeah precisely I mean like when we started this we were doing PhDs in

computer science and distributed systems but we think they're like at least 50% of the product that we've built is just economic incentives its mechanism design really like top to bottom and so you you can make the technical side of it work like in a private sense you just rip out the token but you're missing half the point at least half a point thank you so much for coming on the show and talking to us about this project thanks was very fun if people want to find out more about all of this stuff what should they do where do they go yeah they can go to our we've talked where we have all of the developer documentation a sort small sort of token cannon you can just give us your email address and we'll give you five tokens so you can get started storing some stuff and yeah and like all

of the everything you need is a developer to get onboard with the system is there and what's particularly cool about it is that to build decentralized apps in our network you only require like HTML CSS and JavaScript which you know many many developers know in fact all web developers which make up at least 30% of all developers know very well yes it was very open if you want to get started building that people can actually build apps on top of this oh yeah completely so just before I got here I was building this home in the mail as yeah and all it is is a sort of bundle of HTML CSS and JavaScript that goes to the block we've it's served from the block leaf right so it's got an address people point their browser to it and if they have the web extension they

get decentralized access and if they don't they go through a gateway but it serves back this bundle of HTML CSS JavaScript which says okay give me your wallet let me go check for transactions that are signed to you in the network yeah and then just through JavaScript it displays it as if it's a mail client and it shows you the data stored inside it's really super simple we got the whole thing working in like 6 hours and it's decentralized email which I think it's pretty cool cool thank you very much thanks and to our listeners thanks for listening thanks for listening [Music]