DAPPCON 2018: Building a Decentralized Data Infrastructure - Evgeny Ponomarev (Fluence.one)

Transcript

hello everyone thank you for coming I'm Eve Guinea I gonna talk to you about like how we are built in influence like a query layer for decentralized applications like why it's really needed and what it is and what's like the impression how it works so I would like to start with the example example of our metal search engine for airline tickets and usually when we like use Google flights or Skyscanner on or any other metal search engines they work the same way they aggregate airline tickets from many airline companies they put it somewhere in their database and then

they allow you to source these tickets in order to find the cheapest prices for for your flight and usually they have a user interface where you can specify like city from to like in other many other parameters and you see the search results but like what's what's problem these such services actually they are known not transparent for the user or I can I could give you some example for example one airline could probably pay in Google flights to exclude another airline from search results so and user will never know that because like it's all like a black box this search happens on the server or the

private company owned by Google and it would be cool to create this such a like decentralized version of such Google flights and have the old ticket prices open transparent on a blockchain and allow you perform complex search requests across all of these ticket prices but there are lots of challenges there are like prices are changing so so often the tickets like the high volume of data we have like high load from from user traffic and it's kind of like should be really our real-time performant application in a browser and like traditionally I probably I probably

could should skip these slides traditionally we use like stand database solutions for data processing solutions they all are like they all works because they usually have an index index is like a catalog of of entry points to a data that allows you to find particular entries very fast and efficient and that lets in in the traditional world that that's fine and that works efficient but this like kind of leads us to centralized model when the whole search is performed on a server of particular company so let's imagine that we are building a decentralized version of such meter search engine and like we if you think

about how it works we just need to put all of the data to a thrown blockchain and then we have a browser code a JavaScript that works in a browser that perform a search and takes data from from the blockchain and let's imagine that a theorem solved all of these issues with scalability so we could put everything on on a theorem and it will work fine but actually we do not connect to just a theorem network we connect our browser application to a particular node that either it's like in fewer infrastructure or either scan or miser wallet or whatever or you create your private node and the problem here is a

user that runs the the code the javascript in the browser have to trust this node you have to trust that this node responded correctly and these results was really honest and also if we do not pay to note provider we don't have any guarantees for SLA nobody guarantees us that this node that we connected will be online in the next few seconds so this kind of problem and another issue here if you don't like my example with Meta Search Engine imagine that you would like to display in a browser history of token balances of particular address and to do this you

actually need to do a lot of requests to a theory note and this creates kind of poor user experience for the user because everything works slow it's not like not it's really slow and really bad bad performance and we still need to trust our node that we connected to and the usual solution for this is ok let's let's have our own server let's it style database that we like it will have index that we need and we will be able to perform queries or requests that we need for our application but the problem here is our user still needs to trust the

server our users still need to trust our private infrastructure that connected to a theorem network and it is kind of not a decentralized spirit so finally we have the problem that user needs to trust or am the new provider either it it's sort part you know provider or it's like us our node we don't have any flexible query language to perform a complex search requests like as difficult parameters and so on and we basically have really slow UX and like no SLA until we paid pay for it can we do this better probably yes let's let's imagine that we just replace this

particular node to a network of nodes that connect it to each other they replicated our like database with index and they so we will be able to perform a queries that we need by requesting these nodes and they will be able to prove somehow that this response was correct that would really cool though we will need less trust from from the Europe user using such such Network and we will not be limited just a theorem data it could be like we could use such an index in network to get data from any data sources from many other block chains

decentralized file storage is centralized file storage is any kind of data sources or even we could just put data directly from a browser from one client and read by another plan and also it would be really cool if we do not limit will not be limited by just one query language or data interface it could be cool to have like SQL graph QL and no SQL and other any other custom interfaces that we need having touch network we will be they will need less trust we will have flexible query languages we will be able to create really real-time applications in the browser and they will be more

decentralized and video then when you connect to single node and that's actually what we are built in influence and fluence is it decentralized data infrastructure that based on independent network of nodes that provide like this highly optimized for real-time use cases for real-time performance for the end-users and yeah and so and actually fluent highly like optimized for real-time performance it's fully trustus so in order to use the network user don't have to trust they know that it connected to and because

all of the queries verifiable and I will talk about it a little bit later and like it works with many many query languages and actually like very simple diagram we have Hardware owners who provide the resources for the network and could monetize this hardware by participating on network and store the data and perform the data operations and response to respond to requests from a users they could earn tokens for that we have developers who could implement their own query or like data interfaces or data structures or query languages

and upload it to a network and we have users who actually use like applications in browsers that could connect to a network and get a really real-time user experience and like few words how how it works inside for this in order to optimize the network for really great performance we organize nodes to clusters cluster is relatively like it's you know it's relatively small amount of nodes that well connected they replicate they for example or

responsible for a particular data set and they have fully replicated data inside the cluster and clients connect to this cluster in order to read data or like do a queries and write data and so on and we have the verification nodes who are performed the like the what we call it batch verification for the queries and I'll talk a little bit so the essential thing here is to in order to create such a network we need to have a way to verify queries we need to have a way for nodes to prove that the

response that they provide is correct so how could it could it be done or like at first of course we have the consensus consensus inside the cluster so we need a majority of the nodes to be agree on the results of the query and beside that we like if a client does a request to a cluster on to a particular node it has a response with the merkel proof of response and Merkel proof like is basically a structure that allows a client to verify the this response was correct and if a client have an access to a root chain worthy like root state

of the whole database is third it could verify the wrist once without like any other confirmations from other node from from the network so the client have a few ways to to verify a request first is to verify an oral proof with like this root state of the database and second is to wait until more confirmations from a network from other nodes from like this particular cluster and second thing if the client needs more security we have like the huge amount of verification nodes who verifies request that like was

performed by a cluster in a little bit like a delayed mode so they in one in certain period of time they were if I the queries in in very similar like using similar mechanism as I explained in one slide before and they like sign the that this request was performed correctly or incorrectly if the request was incorrectly the cluster is being removed and of course like in order to have the like the honest behavior on the network we have the security deposit so

in order to store some data on the network you have to create a smart contract between you and the node or in a cluster and like a client and the cluster have to put some security deposits on that and if the network is like seed that this request was not performed correctly or this data is disappeared so we have an insurance for like both parties that they will get their money back and and also it's important for nodes because they perform a work and like it's also guarantees that the client will pay for the work that was performed correctly and finally

like if we build such decentralized metasearch engine we would probably use a theorem for the financial transfers and we will need induct the fluence for data queries and search requests and performing the computations on top of data that needs to be it needs to be displayed in the browser in in a second so just momently and I could show you a quick demo it would be it's better to show it here in a browser yes so let's let's try I can I'm whatever time do now I'm it's kind

of could be problems with Wi-Fi so try again I'm generating keys just keep air to connect your network and yeah now I'm connected so it's a private private test Network it's like a small notes that operated only by us and now what I did I open in the web page it has a JavaScript and I connected directly to to the network like without any trusted nodes but these nodes of course afraid it by us but it's temporary it could be like any nodes and we have the first implementation of key-value interface

with range requests and we have the like the data is organized in mortal be trees on on the nodes so when we do requests when we put data or get data from the network we receive merkel proof that it is like this part of data was put in in a particular place of north north will be three so let's try to read something here probably I could not read anything now because this keeper that I generated was new and this contract storage contract is just created so but but I could put something like some data there let's put just a few strings seems like

the connection is not not so good still still trying to get a response probably probably I could could do again it just like I don't know the device I always on the conference's is kind of challenge