Swarm

Transcript

[Music] all right so um we're going to be listening to a presentation about swarm by Daniel and Victor um swarm hasn't really had as much publicity as it probably should have um so please welcome these guys to the stage they're doing awesome work thanks guys I start the talk right now before we set up yeah just so that I don't want to make the same

mistake okay so so so what is swarm after all so swarm is supposed to be the for for Content distribution and storage for for web 3 so basically it's it's the content distribution Network for uh for the ethereum ecosystem uh so if if ethereum VM is the is the word computer then you can say that swarm could be the the word hard drive or a word hard drive because there might be alternatives to to this solution so on on the other hand like what else is is a swarm I mean you can you can can just imagine an imaginary swarm where like the the happy

beekeepers like sit behind their hives and and look at their fantastic product of the from the bees and just happy happy to have their honey harvested from The Beehive you can also look at the Beehive as this lovely repository of uh honey where all is like drops of Honey are stored and also look you can look at that um swarm and and see the the buzzing swarm of bees just communicating with each other I don't know how you are but like for me it obviously reminds me of of of a network protocol and like all the network messages I sent around so let's let's concentrate on these three aspects so uh so just go go

back so uh so so first of all um as as as a service what what swarm provides is basically it it allows to to store the the the ethereum depths the decentralized applications and also kind of offchain uh data that that's important for example contract um sources or like registry indexes all kinds of things which need to people need to store and what what they want for that like users want want to keep this uh record safe and in general you want they want Integrity protection always availability like basically zero downtime for tolerance all these nice things that you want from a decentralized solution in general U and

one important message to take away from this and they are willing to pay for this in a way like the enence is there for you like the user is wants to have the contract ret the the content retrieved and that's that's an incentive so uh what how we how we solve uh this problem is is an interesting you can reduce this problem of file file storage and basically hosting to uh to to to to resolving the problem of of storing fixed size chunks in a decentralized system how how is that working basically on the file level or document level you can think of the the you can basically chop chop up the files into small little pieces rearrange them in a nice Merk try

this is not a swear word in this company of people I mean you all know like the ethereum works in the same way the ethereum state for the Patricia try and what you end up with this uh with this merker try is that you can have like a random access partial read of a into a file with still preserving the the integrity and you can you can do it you can do this like one step further and say that you don't you you can not only handle like documents but also document collections so another level of indexing which is kind of similar structure you call them you call it the Manifest and this also allows to index but as opposed to the previous Merk TR the the parts of

the the the document collection obviously artific arbitrary Lang length and the the indexes are are just pass so this means that if if you have a root of a of a manifest like this this chunk then basically they can represent entry points to uh to to the to the web so basically they can be thought of as hosts because the because the Manifest is basically the uh constitutes the rooting table for that for that host so basically you can use these chunks as as representing or or mimicking a a web CER Ser so this is how you how you make like a decentralized server from from swarm now so basically the message is here

that uh we we reduce the problem to a to a chunk store and we associate each chunk with a with with a with a hash hash of the like the hash of the content which means that we we have to look up them based on these keys so what what do you actually store let's let's look at the other aspect of the of the highligh the store so each note has a has a memory store and and the and the dis storage so basically these are the caching that the the note can look up uh locally but eventually like there's going to be a situation where you you can't store everything so you have to like look out to the network to find something because the the the neighbors

the neighbors honey is always sweeter anyway so uh now now you have to to solve this problem of of how how you communicate your needs how to find basically these these chunks so what what you basically do is um is use a use a network protocol and uh swarm relies heavily on the ethereum network which is basically Alex lington and Felix lung's wonderful you know genius product it's a it's basically a the death beer to-peer multiprotocol Suite I don't want to go into the details just refer you to Alex's talk on like I think it's Wednesday morning uh the the most

important message here is that it comes with all the beautiful security and integrity properties of this network implementation and also it has it gives you a semi-permanent uh peer pool basically the the the peers that you actively connected to can be considered relatively stable and also the network overlay topology is is flexible so all we needed to do for swarm is basically use use a cademia routing table cademia Pier selection mechanism and we immediately had like a a key based rooting system available now how how you combine these two ideas obviously if you if you um

establish that the way you find a particular chunk is to go through like the this the cadamia rooting sorry for for those of you who are like not into this Mambo jumo like let me give you an example so the C routing basically would be that you organize your hive so that for each chunk uh you can find a b that can take you to a hive that's closer to the to that uh uh Target than you are by a factor of like a fixed factor in kind of and basically this logarithmic distance that that the keys space defines uh in um makes sure that you can find a particular either node or uh

information about that particular Chunk in a logarithmic time in in a in a in a fixed constant number of hops so this is basically the idea of DHT it's that's nothing not much new here the the interesting Twist on the DHD that we have is comes from several properties of this of this SE semi-stable peer U uh base and but more importantly the other thing is that we we don't only assume that that the nodes um that we reach contain like give you information about where the chunks are found but they actually give you the chunks themselves so it's a proper uh content addressed storage what's the uh advantage of this

this view um this gives you U the possibility to Define uh a Content push mechanism basically basically how the distribution of content works not not only their retrieval uh basically along the same kind of lines as the routing works or even simultaneously to that and one consequence of that is that is that eventually the um as as a result of continuous retrievals and the the the scheme that we define basically this through this forwarding uh the requests and forwarding um um deliveries the the the end result of that is that

you end up with the with the with the Swarm being a kind of elastic Cloud which means that all the the content that that is popular can end up relatively far from the from the node that's supposed to be close to it supposed to store it but that means that the content will be uh more readily available in less less hopes because it's it's just more per pervasively present in the network and therefore you get an autoscaling that popular content will uh will be retrieved with less latency or that's that's the idea and also this this this means that SW is basically a constantly maximum utilization system so so this for forwarding and and the distribution

scheme that we also call like smart synchronization it also makes sure that uh the the network adapts to like when when content becomes popular it basically replicates and you think you can can think also of of it like a like a giant donut that grows when everybody everybody bites bites to it and and and the same way it can also shrink so if if if document popularity is doings then then it's it's it's continually deleted from uh first first those nodes that are relatively far from it for for which they are the popularity doesn't justify anymore that they store it U just

because they they they they are too far and and and so you can think of it as the donut shrinking and in the end it can also happen that certain content is is deleted now let's go to the incentives that's this kind of fourth aspect of the of the uh of Of Swarm so each each of the decisions that um um the Swarm participant nodes uh take should be somehow um legitimated by by some sort of incentive because as as we all know they are like rational actors and they are not shying away from maximizing their profits so they can abuse the system if we don't

align the incentives properly so what what do you actually store let's go back to this question first you you we can answer this question simply by like you store what you actually deem most profitable to store uh it turns out that all you need is to consult like your database and see how much a particular chunk or content was accessed it's it's because actually on one on one hand access is a very good predictor of future profitability of a chunk to assume some sort of um constanty of of of popularity over time at least and on the other hand it's it makes sure that the that the um content um synchronization uh results in a

network where where which which behaves like in this in elastic cloudway so do the Autos scaling now uh I I I I realized that uh the incentives for retrieving certain um content takes care of only those cases where where we talk about deps like this typically popular content that you want to distribute and you want people to to to reach this is not all the cases we also invis Envision SW to be used for cases where where you want to store and actually make sure that you your content is stored even if it's very seldomly accessed for example a birth certificate and we realize that this incentive that

we we talking about is not handling that properly among other things it's because it's a very different type of of of reward mechanism because here here you actually want to make sure that my birth certificate is is is available I don't want to pull it all the time because it's anyway insecure and inefficient so if I always receive it so it's a different different system it needs there's a lot of discussion about this I I'm not not going to go into that I I just refer to the to the documentation and like the the kind of orange paper that we we're in the in the middle of writing for for this discussion uh the the main point is that one type of incentive that we do talk about is is

can appropriately handle the the incentives for for for retrieval and basically which also translates to uh the pricing of bandwidth basically because that's that's that's what it does as opposed to like dis storage uh now for this we we're using the the the the swap system which swaps just is a kind of pun that that that you basically swap payments and and service but on the other it's also abbreviation of swarm accounting protocol and some other things and the way it works is um is is that basically it allows you to to account for for in general uh kind of

any commodity class that can be um that can be Quantified and has certain other properties that that you can use this for and swarm it's it's used for for the for the accounting of bandwidth and uh in in in swarm you use a a checkbook uh contract on the um um blockchain and that allows you to to to do the delayed payments that that this system uses so it's basically you you you it's it's a it's a micro payment system in which you you you use a pairwise accounting for for peers where they can exchange one unit of service from one unit of service

so that's cons if if they consume and provide the same amount of service then their balance be is hovers around zero obviously there's there can be like things that if I if I work at night and the other person like you know download films in the in the in the in the in the day there can be like vast differences in the utilization over time so for these cases basically when when the balance tilts towards one particular peer then the uh then the the peer is obligated to send the compensation which is an offchain delayed payment in this case it's a signed check the signed check in this this type of offine offline payments can be verified offline

but in order to redeem the the amount that they are about you need to transact with the blockchain and the transaction cost is is is burdened on the on the recipient which means that they are incentivized to delay caching it and thereby save uh blockchain time and blockchain uh cost and also they are indirectly incentivized to watch all the the kind of um aspects of their peer which make them trust them more so that they can delay the payment even longer so basically it incentivizes the emergence of of a reputation system in a way and that's also possible because the checkbook contract is basically

historical it's a credit record that you have so basically at the end of the day uh this this system can uh all like to to to trade reputation for some sort of uh up deposit and uh most most importantly um uh can incentivize the the the the the retrieval and correct working of the network now just just a quick summary before I give it over to uh to dania who will talk about the the upper layers and and some um some of these the applications for for deps so just to summarize so basically swarm is defined like more formally as a as a

protocol the is that protocol uh which is kind of a a mix of several different aspects of of of the of the of the system so basically it's also a protocol for for for peer um selection and peer um like basically bootstrapping your your your node and connecting with with others in the network it's also a protocol specifying the the the request forwarding and how how the how the retrieval is and then the routing works it's a it's a synchronization protocol and and also it's a it's a swap protocol it's like the the payment it has maybe maybe a fifth part which relates to the storage

incentives that we don't talk about that's that's also part of this protocol that the communication here and of course it's it's it's also defined by a set of apis uh which are more related to what um Daniel is going to talk about so um I give it over to him thank [Applause] you so hello uh I'm going to talk about what can you build upon this solid foundation of a incentivized chunk store so once uh we have in place a reliable retrieval and storage infrastructure

that stores fix siiz chunks we can build uh uh web-based distributed applications on top of those and I'm going to talk about these possibilities and the apis that we provide for application developers so the most important I think is the HTTP based API which uh really acts like a locally running web server uh with the guarantee that the usual uh methods the usual HTTP based methods are all efficient so you can change single files you can retrieve uh via range queries uh some portion of the

file which allows for uh streaming and you can change uh uh the content of your virtual web server uh which of course raises the question that this whole thing is content addressed so what actually happens when you change something so here are a few URL examples in the first case you're accessing the raw API which means that you're just downloading without any metadata a large set of bits or bytes and then uh the next one is a swarm site which is a root page of a web server I actually have it here

uh so it's it's it's here this is a photo album uh so this is a photo album and if you click around then you can see that there's a part of the URL which is just the uh the fragment part which is not a state that you want to share it just which picture are you looking at so this is a state that only exists locally however if you delete uh a certain picture that means that as you can see even the root has changes and once you share this one then the uh the entire content

changes and if you rearrange the pictures or you upload the pictures the exact same thing will happen the root the root has root hash changes and uh therefore you can share that state with others and you can do that through a name registry contract so for for example this URL here clipped Wings would be always the current state of this photo album but this uh so the Nexus with ethereum blockchain is first of all the Swarm naming system which is just a single contract where you can register root hashes for uh for for names then there's this content

availability Insurance where you can put up a deposit for making sure that a certain contract is uh certain content is available and If somebody challenges You by saying that this particular chunk that you promised is not there unless you upload that you will lose that deposit and possibly the person gets rewarded for uh for the trouble uh then you can also store off blockchain State so uh this refers a little bit back to the uh scalability talk so one way of thinking about somewhat scalable distributed applications given a traditional blockchain is that you treat the

blockchain as an Arbiter of disputes and as long as you don't have disputes you don't consult the the the consensus the uh bantine consensus of the blockchain but if there is a is a problem a disagreement then you resolve it through the uh through the uh blockchain and there are all these other things which kind of Link swarm to to ethereum uh so the uh distributed application the photo album that I have shown is basically the simplest distributed app because you have one producer many consumers so you kind of Sid step the

difficult problems there's no concurrent modification changes don't happen frequently and uh latency is accept is acceptable like if others see your photo albums changes after a few minutes or even hours that's not really a big issue however it's interesting to think about how would you go from web 2 to web 3 like how do you build complex concurrent uh uh distributed applications and we think uh and we hope that swarm as an infrastructure together with ethereum is actually suitable for this purpose so in case you want to do something like Facebook where

multiple uh users can comment on the same content then of course you need to con to to uh transact with the blockchain more often and to query the blockchain more often to see what the actual status of a particular content is and you also need something like whisper to distribute uh real time uh commenting information uh building a index of uh of swarm well that's a really difficult task and it's you know it's a topic for probably a separate talk or even a SE separate section but the basic idea here is that you take the uh map redu Paradigm with untrusted

workers and you specify what they can and cannot do in a contract and again in case of a conflict when they violate the contract then you can punish them and otherwise you can reward them now the other interesting cases I think Wikipedia and open street map which are examples where the web 2 model doesn't really work very well and where we can actually substantially improve uh on the state of the of the web because in these two cases there's not a real world consensus about the content uh so in Wikipedia you have anything related to re recent history of conflicts where there are edit Wars and there are you

know disagreements about what should be there and in open street map you know it's a current question whether liberland is a country or not and uh therefore uh like forking these projects is not really feasible so people try to do something like conservapedia where they try to eliminate the supposed liberal bias from Wikipedia but the problem is that they have so little content and it's so expensive to Fork over content from the mainstream Wikipedia that that project is simply unlivable but with with with swarm what you can do is you can present different views to the same large body of content and simply mark up uh like which point

of view would you would like to take and it's really really cheap to make slightly different versions of the same huge data set so I think that in case of maps and encyclopedia we're actually looking forward to something that is qualitatively superior to what currently exists in web 2 uh so finally I would like to tell you what of this exists and what's the status of this project so we have a reference implementation of uh so this is an extension of the go client gu uh which has a uh uh HTTP which has the HTTP API it has a uh the uh pairwise

accounting uh all of it works on small scale uh we simply haven't been able to test it on on a larger scale but uh and I'm sure that it would have some problems that need to be uh ironed out uh so far it has been funded uh by uh ethereum foundation so this has been uh and we have had a lot of Community Support as well uh and here are the things that we are ready to demo well not right now of course but uh hopefully during this conference and in GitHub you can see the source repository which is basically a

fork of the uh go uh ethereum client with a separate Branch Bzz where you can always find the current state uh of the bz capable uh G client and we are keeping it up to date with the uh Upstream version of go ethereum so uh that's where you find uh our source code and uh our documentation as well uh maybe Victor do you want to say anything about the status okay than thank you