DAPPCON 2019: Workshop - Building Fast, Decentralized Web Apps w/ Sam Williams (Arweave)

Transcript

who here knows what our weave is no one okay fantastic so our weave is a kind of permanent information storage system the idea is that we've taken a blockchain and made it so that you can fit large amounts of data inside it and miners we incentivize to store that data over extremely long periods of time from a technical perspective it uses this plot this data structure that we call a block weave rather than a blockchain the idea is that instead of including just the hash of the last block in the production of the next block making sort of a cryptographic chain structure you also include the entire contents or randomly chosen previous block from the network in the production of the next the next block and this means that if you don't

have access to that data you can't generate the proof-of-work challenge required to take part in the production of the next block so you incentivizes a miner to store these old blocks in fact you're actually rewarded proportionately to the number of those old box you store creates a kind of incentive to store the old data in the network so at the base layer it's just a big on chain data storage system for preserving data over tens or hundreds of years on top of that we also have a data distribution network which essentially is kind of like optimistic tit-for-tat from Bitcoin it's sorry BitTorrent if you know it the idea is like if you give me some data I'm going to give you some data and then occasionally we just will give data to each other randomly and see what happens

and because of the way that the the mechanism design in this works it tends towards an equilibrium where we're all sharing data with one another all the time yeah so there's a sort of base layer for storage and then another layer on top before the distribution of data so the entire system is built on top of HTTP and web technologies the idea being that you can then take a transaction ID from the network and you can put it in your web browser and you can just say hey are we've node X yeah please give me the data associated with this transaction and that transaction will contain normally some kind of webpage and it gets rendered in your browser so essentially what you have is a web built into a blockchain and this works very well as a kind of hosting environment

for decentralized web applications and particularly web application you is the if you are building a smart contract say you typically need a UI to go in front of it but the problem is if you put that UI on some kind of centralized service at some point it's going to be lost and forgotten that doesn't work very well you've still got the smart contract in the background so let's think about Krypto kitties say you still got your crypto kitty but if there's no UI in the front-end to look at then then your crypto kitties are really just numbers so the idea is you put the UI for this system yeah onto our weave where it's permanent and then it's just as permanent as a smart contract and the two things sort

of live together and so the user has access to the application for much much longer okay that's kind of the base level of my presentation that we couldn't get working on this projector yeah let's just try deploying our weave page so the first thing you should do is go to fact I will look so I can do it as week as we go as well yeah all right so if you go to docs.google.com then this is everything you need to get started building web applications on top of our weave in fact it's really really super simple there's basically nothing

to it so if we click our weave deploy user guide it takes you to the page that's got all the information you need to build web applications yeah if we click our we've deploy user guide our weave deploy is a really simple CLI tool that just allows you to take a HTML page or some other file and then dump it into the blockchain is actually yeah so the first thing you need to do on a terminal is NPM install - gee our we've - deploy install the deploy tool it's just this yeah and there's just give it a try so directory

is that I'm assuming that's too small okay so let's just make a simple web page Wow the color of the terminal changes the room come here so we'll just make a super simple hello world hello comm and of course this can basically be any kind of HTML you like and so if we

have the deploy to installed we can just say our weave and then a command like balance for example and it'll certainly what many provinces if you haven't set it up yet you can just say yeah you can load a key file which we'll get to in a second so the critical thing is you just want to say are we've deployed and then the name of the file you hit that and decrypt your key file if you need to and it'll tell you okay I'm gonna make a transaction called tspn bla bla bla bla bla it's gonna cost basically nothing it's gonna be deployed with these tags and it's gonna go from this wallet yeah you say confirm there we go now that's going to the our weave and if we go to

this address once it's confirmed so literally written into a block yeah we will be served back the HTML page that we just created so one of the interesting things about our weave that's kind of different from other data storage systems like ipfs or dat or sia for example is that we have a slow write fast read approach so you you're always kind of making trade-offs here and normally the trade-off is that you can write data into the system very very quickly but to read that data out or more specifically to find that data it's very slow we take the opposite approach because we think that web developers when they deploy applications they don't mind waiting a

small amount of time to do so if users can then access those applications in like sub second or sub 200 milli second times so you'll find that once the thing is actually onto the system you can always access it extremely quickly and your users can access it extremely quickly but getting it there you know it takes a little bit of time yeah okay so well that's confirming we'll get what I guess so who has installed the deployment tool all right a few people well if you're following along and the next thing you need to do is get some tokens the easiest way to do this of course you can you combine them you could we could also just send them to you and yeah you could buy some but the easiest way is if you

just go to tokens dot R we've dog which has this sort of embarrassingly centralized continue with Google button which allows you to claim a free token and we use Google because it's actually reasonably good civil resistance for identities it's kind of hard to forge like hundreds of thousands of Google accounts and yeah so if you just click this you have to Train Google's neural network for a moment and in fact I've already got one so it's not going to work for me yeah but if you do that then you'll get sent an email you just press the button and you can download the key file and then you can use it to deploy pages and as you saw before the pages are extremely cheap to

deploy so it doesn't take very long this is really the core of what our weave does and when when when it's confirmed it looks a little bit like this so we've watcher is a decentralized web application that's all made that just looks backwards through the block to even sees what's stored inside it it's really simple it's just a HTML page and I think this cost I could find out actually so we take the transaction ID if we look on view book the block Explorer we have see this was submitted four months ago and it cost zero point zero zero zero

zero zero point zero zero zero seven R which is yeah a tentative percent and the cool thing about that is now that page is deployed it will just always work there and there's economic incentives and mechanisms in the background to make it so that its profitable for - to store that data essentially indefinitely that's actually kind of an interesting mechanism which and as well get into well you guys are getting wallets and things yeah so when you put a piece of information into our weave you pay upfront for 200 to 300 years worth of storage and those tokens get put aside into a pool and over time as the cost of storage declines the

purchasing power of those tokens that you've already put aside increases essentially this means by the end of like year five or ten you have tens of thousands of years worth of storage pricing already like backed up and as long as this rate exceeds around yeah the storage cost decline rate exceeds 0.3 to 0.5 percent just never stops it's basically like an endowment so if you had a student endowment you would say okay I'm putting aside ten million dollars and then I can take away like you know nine thousand dollars every year indefinitely to pay for some of these tuition fees same kind of principle except with storage and we get interest from the declining cost of storage rather than the interest you

normally get on a financial instrument yeah so does anyone have any questions somebody yeah well the no donor is expending value right to store it that's true but they're being paid by the network and it's not like one person is assigned the storage of the file it's actually more that there's an incentive for everyone to try and store the file in fact they're really competing to store the file in the same way you would compete in Bitcoin to have the largest amount of hashing power or really you're competing in a race to find a candidate box first sure I mean yeah yeah it has

the incentive layer on top it uses different kind of routing ipfs is really just a way of saying I have a Content address I would like to find it somewhere in the network it's just a big DHT we actually don't even have a DHT yet the way that we look at it is finding content in a decentralized network like a really big one is an extraordinarily difficult problem like people have been trying to do it since the 1980s just no solution so far scales with ipfs like we built an extension that takes our weave data and makes it available in ipfs so you can also take ipfs data put it into our weave and then make it available in ipfs the idea being

that we can apply the incentive layer that we have to data and ipfs and so this means we played around with ipfs a lot and we found that like on average the data access time is seven minutes on a public high for your first network it's it's pretty crazy so and this is not fantastically surprising like people have been looking at this problem for a very long time and condemned nia which was built in like 1998 for the Nutella project is the best attempt we have at addressing content in this manner in decentralized network so we take a different approach which is to say okay well if finding data like the needle in the haystack is hard then why not flip it why not make it so that the data is almost everywhere as a result of incentives so that finding that data is

extremely easy and that's essentially what we do with this proof of access mechanism which is enforced by the block weave structure because if you want to take part in mining you have to replicate that data and the consequence is just way way easier to find it so in the network at the moment if you go to yeah let's have a look I think we've got like 500 nodes or something and the average replication rate will be yeah the average replication rate before say this piece of yeah in fact looks like our our page has deployed so we go to we've done net and sure enough hello yeah our page is now mirrored on 500 different nodes so

if you go to a random node in the network and you say hey can you give me you know whatever this transaction ideas there's a very very high likelihood that they're actually gonna have access to it and this means that you can yet you can have decentralized web application experiences that don't suck for the user because at the moment the alternative is that like they go to your page I mean we could try let's see if we get lucky I think ipfs dot IO is stored on ipfs and like it's super slow yeah it's I think the the best way of doing this that actually works for users so it's pretty simple you just put your smart contract that kind of system onto a

theory or more have you and then you just build a normal web UI and you package it up and then you deploy it on our wave and it just works yeah yeah so suam isn't live yet so we can't really see what the performance is like in the life Network but it will presumably be around the same as ipfs or dat or really own Nutella or Napster or any or kazar or any of the other lime what was that one called LimeWire I think it was like any of the other p2p of file sharing systems that have existed previously it'll have the same kind of performance mechanics because

it's based on the same algorithm and in terms of the incentives it's called that swarm of building incentives into the system I'm really interested to see where this goes actually because the reason that file coin didn't use etherium as far as I can tell for there yet the data storage incentives in the network was because if you have you you're trying to build a decentralized web right so it's going to have billions or trillions of files if each of those files has a contract associated with it how are you gonna get etherium to mediate that and the file coin solution to this as far as we can tell from their github repos is basically to say okay well you just don't adjudicator all of the contracts so if I put a piece of information into file coin it's then my

responsibility to check with the miners that have been tasked with storing my file that they are actually storing my file and we don't do this at a networking force level because you just wouldn't scale so I'm really interested to see how swarms solve that problem well I think we take a very different approach and we're not really trying to do the same thing so I think there's like decentralized storage networks in general that are looking at maybe really really large files that's not something you want to do on our we've yeah I guess we're just making different trade-offs

so they're looking at like Chuck can you build an interface database yet you actually have lots of applications that do database like things so I can remember the address now but there's a decentralized kora system on top of our weave there's this thing we call our QL which is a way of so every transaction in the network we need back of this can have tags and there's a database on each of the nodes that just keeps track of the tags of each of the transactions it's aware of so yeah you can essentially query that database and

find all of the transactions that are associated with for example the application time for looking at we actually think in the longer term so this is this is not a replicated database is just on each node so you're talking to the node about its view of what it knows of the network we think in the longer term other decentralized database projects will come along and you'll essentially submit your data to our weave and then you'll submit a record to this decentralized decentralized and globally consistent database it says hey I've got this always piece of data that's associated with these kinds of record or you know these tags and then you can query that and it gives you a globally like consistent view but RQL works really well for now so you can actually build like interactive decentralized web

applications with alright any other questions okay thank you for this very weird and dark workshop this morning I hope you all have a great day oh well I mean yeah yeah exactly like yeah if you just make a replica of the

entire web and you put it in a data center and then you access it sure it's totally gonna be fast that's true but realistically most of the data isn't even replicated inside it I don't know why but like the CloudFlare ipfs cache isn't really that large which is surprising you'd think they would just basically say well let's just store it all and part of the problem there is like you can't find it you can only find a piece of data in ipfs if you know what it's called so for example we're working with this group called every PDF one of the Wikipedia founders spun off to make this sort of Wikipedia that was more decentralized and less kind of like less control over what articles should or

should not be written essentially so they have all of Wikipedia in there to start with and we started saying okay well we can grab the data from ipfs and we can put it in our weave and then we you know do this thing I was saying where we also exposed it over ipfs again and what we were finding was that like even though without flair could store copies of this and we were accessing through the cloud layer database oh sorry the cloud for their endpoint to try and make it faster even if they could store copies of it they weren't and so still taking like seven minutes and sometimes when you go to ipfs dot IO even which I'm pretty sure uses the cloud for there yeah the cloud for the Gateway sometimes it's forgotten it it's just dropped it from the cache and then you have to

actually go out into the real light PFS network and try and find it what we're seeing more and more is that people are using IP FS in local systems where it works really really well it's great for that so you have like a small local area network with a whole bunch of computers and you want to just like share some files around ipfs is fantastic for that it does the DHT that works really really well it just doesn't appear to scale to global sizes so yeah a couple of different things like things like this

where we can talk to crypto users that are building gaps and they need somewhere to store the UI it's a really simple and obvious use case and it's fast and there isn't really an alternative right now which is usable you know you can put your interface on ipfs but from the people that we've seen and spoken to they do this in the background but then really they're just using AWS to serve it cuz you know you can't have the user waiting seven minutes to get access to the webpage just they're gonna go and do something else and on top of that the reason we got into this was to make censorship archive of history the idea being that you know if you're familiar with 1984 the idea of the memory hole right that you can essentially if nobody remembers what happens in the past it's

impossible to really tell and if you don't have records of it it's sort of in the ether and that means it can be manipulated and that changes the way we think about the future it also changes the way we make decisions in society so we got into this to build a solution to the memory hole yeah and now one of the paths to adoption we see is just making sure that the tools are in the places like for example in Hong Kong right now where bits of history might be censored and then making sure that people are aware that now there's an immutable record of that bit of history you know we're experimenting we'll see how it goes Thanks