Yaniv Tal: The Graph – A Marketplace for Web3 Data Indexes Based on GraphQL (#283)

Transcript

[Music] this episode of epicenter is brought you by cosmos cosmos is building the intent of blockchains an ecosystem where thousands of blockchains can interoperate creating the foundation for a new token economy if you have an idea for a DAP visit cosmos network slashed epicenter to learn more and to get in touch with the cosmos team hi welcome to the epicenter my name is sebastian cuiture and my name is Frank Fontaine so today we have Yaniv Tala as our guest

yaniv is the project lead at the graph you may have heard of the graph it recently launched and the graph is is interesting because it solves pretty prevalent problem in the a theorem space and that is the centralization of etherium no I'm being facetious here but as many of you know gaps that are building on aetherium are having to rely on index data so the blockchain itself isn't actually built to you know sort of index data and have that data readily available for taps and so what many dot what many gaps are doing is that they're building quite centralized infrastructure on servers and allowing

adapts to query that it that that infrastructure with traditional web to api's and so this is obviously a problem if if the goal of ethereum is to build the world's computer that's totally decentralized so the graph is taking an interesting approach and that they're building a graph database on top of the theorem as a layer on top of aetherium and we sort of go into what a graph QL database is if you're not familiar with that technology and allowing gaps to query that directly so making it much more efficient and so their goal is to have this become a decentralized market place for index ease of D theorem dip B theorem blockchain

so it's it's an early project I think generally we try to wait a little bit longer in a project maturity for to bring the the project on the show but in this case we just thought it was such an interesting project and felt that it was a good at least now a good point to have them on and perhaps map on again in the future when the project fully goes live and is fully decentralized so last week I solicited contributions for the Wikipedia article that solicitation is still open so if you want to contribute to the Wikipedia article that is in draft you can go to episode about rocks slash wiki and you'll get directly to the article another thing that we should mention and if you're hearing this

that's good news because we've updated our podcast hosting service which means that our feed has changed and so your podcast app should have reset your feed so you should be hearing the new feed one thing you'll notice that's different in the feed is that episode numbers are no longer in the episode title and that's simply because modern podcasts players have a native field for episode numbers so you'll you just won't see the episode number and the title anymore if you're not seeing the episode numbers and you want to see them then I would recommend you know updating to more mature podcast apps so like the iTunes app supports them pocket cast supports

them that sort of thing so just want to mention that so without further delay here's our view with yaniv top so we're here today with Yaniv tall and he's the founder the project lead of the graph now we're gonna speak with a Neve about you know a lot of different things about querying and you know probably one aspect of the DeSantis application web 3 stack that a lot of people are not as familiar with so thanks so much for joining us today and Eve ladies what for happening oh maybe where we can start is if you can speak a little bit about how you became interested in blockchain and you know have the perspective you

brought to blockchain sure so I first started getting just said in the idea of blog chains in 2011 I think you know following you know the Occupy Wall Street movement I became very interested in thinking through how we could improve governance and the economy through digital money but I didn't actually get it make it like a you know my day job until 2017 so a lot happened in between I spent a lot of time it's different startups I did my own startups with my co-founders here at

the graph and focus on developer tools for quite a few years and the perspective that I took into blockchains was trying to figure out how to make it easier for people to build software just in general so we had a start-up what we were doing react developer tools to make it easier to build user interfaces we got into functional programming looking at building on immutable data and and an along that path what got me really excited about etherium was the idea that for the first time we could have this kind of global immutable database that isn't just bounded by a single

organization but so that everybody can agree on a global state of truth and that to me was just really exciting as being like a foundational primitive for just how software development could move generally what was interesting at listen to another interview you did and it was interesting to hear you speak about blockchain kind of from this perspective of okay developer efficiency you know because generally when people speak about developing blocks in applications everyone complains oh it's so inefficient so hard to build and you know there's these other benefits like maybe decentralization and you put up with all of the you know the hard time of developing something because there's

other things that you know make it so much better and now you kind of look prospective Oh actually it can make it easier to develop so that I thought that was super interesting here that's exactly right and and I think that this immaturity is just because it's a brand new platform and you know we've spent like 20 years building up the web platform to what it is today so at the beginning people were building their own servers the web servers and like C++ and that wasn't easy but they put up with it because you know the the web was exciting for them and and so similarly I think you know we are in the infrastructure phase and and um you know I think as we build out more of

this infrastructure the it there's no reason why it wouldn't be even easier to build on web 3 than that then it is on web 2 and I'm quite confident that that would be the case but there's there's plenty of work that needs to be done before we get to that point so why did you guys decide to start working on the graph um well we got interested in etherium seriously in early 2017 and we started you know building different daffs and would quickly stumbled upon this problem that it's actually really

difficult to get data that you need to power a web or mobile app directly from an ethereal node you know I think we got into the space probably for a lot of like idealistic reasons the same reason a lot of other people got drawn to the idea of decentralization but the graph specifically was basically partially realizing that there this indexing and query layer is completely missing from the the web 3 stack today and and we've actually spent a lot of time working on this part of the stack at previous companies so for example at our last

startup we built a custom framework on top of an immutable database called atomic and we Bill you know this graph QL like query language on top of this immutable database and we just did that kind of out of passion just because you know we're always trying to refine you know what's just the best way to build software right what are the right abstractions just for building applications and and so you know we'd already spent a lot of time thinking about that before we got into ethereal and then when we saw that this part of the stack was missing for web 3 one thing I thought would be interesting to

speak a little bit about and that that I thought was really fascinating is the way you guys subscribed kind of the problem of you know the way databases and api's work in you know web 2 and you know the kind of effects of that such a view rigidity of api's duplication of data data and effort can you explain a little bit you know how that work what's the status quo here and what are some of the downsides of the status quo yeah so you know today every web 2 company is basically running a fully vertical stack right they manage the infrastructure the

database the application you know the user interfaces and you really have to you know trust these companies to kind of you know continue to manage this data and make it available to their applications and you end up with these kind of data silos where you know that the model for web 2 is you know you build some kind of you know process around some data and then you restrict access to that data and you know that is generally called like a mode and you know how well you're able to build a

moat basically determines you know how successful of a company you are in web 2 and you know I think that's just very limiting for you know enabling innovation to continue and if you kind of look at this larger arc of you know software development it's taken maybe 20 years to really figure out how to build applications that people really want to use on you know even just on the web and you know we've seen the transition from you know PHP to you know Ruby on Rails to client-side frameworks and micro services and data science and all of

these different disciplines where we're just figuring out how do you actually deliver software at scale globally on the web and you know the companies that you know figure this out we're able to you know build these moats and data silos where they control access to that data but we've kind of gotten to a point now where first of all these are more or less solved problems and so people now know actually like how to build false talk web applications that scale globally and yet we've continued to have these kind of you know monopolies that

now kind of control access to that data and so it seems to me that the next kind of phase here is to basically commoditize what you know we've collectively learned and to actually you know make that data more generally available so that we can continue to make progress and then the thing that you guys subscribed F what was nice knows is if you have like the current paradigm where you know I I have some data and maybe I make an API available but then I have to decide you know what what gets exposed in what way and so I have to somehow anticipate what people

are gonna try to build with it right and then if they try to do something else it doesn't really work alright so yeah so that that's a big problem with with wrath so basically the way rest is kind of evolved is initially the idea is you have one endpoint per resource and that way if you have you know some interface that requires you to combine multiple resources together then you can make several round-trip calls to each of those different endpoints and that's how you get your data and as we try to build applications that load really fast we realized that you actually can't make all of these round-trip requests because otherwise you're going to be staring at

a spinner for too long and so what companies have ended up doing is building custom endpoints for their different user interfaces so that you can load the data that you need really quickly but the problem is that now you have to basically you have this tight coupling between that one API and the client and and so if I'm building a new feature suddenly I need like you know a new field to put up in the interface I have to go talk to the server engineer ask them to add that field and it really slows down the pace of development and so you know Facebook is a massive company they're constantly adding

features and building new products and they like everybody else you know was grappling with this limitation and because of that they invented graph QL which is a query language that is really built with clients in mind so that you can specify a schema upfront on the server and then the clients can request exactly what data they need and get just that data back in a single response ok that's that's a good explanation I think one way of looking at this issue is that you know api's themselves were sort of data silos so for every resource there

is a data silo if the company building the API has actually need to for that silo they'll build it into the API or if they think the users will have need for that silo they'll build it into the API so what I'm going to look at this is to say ok if I need to know the data relative to the organization which is attached to a user well first I have to make a query to the user endpoint which is kind of a Cylon itself and that will return the data with the user so name you know bio photo etc and then an ID for an organization which I then need to query to retrieve the information about that organization so in this case it

would require two queries whereas with something like graph QL you could do this in one query so at this point maybe let's let's dive into graph QL and you know how how that sort of changes the paradigm with regards to SQL and how good doing things for the last you know 20 20 years plus yeah so graph QL is a query language that is really built with application developers in mind and so you can define your schema which is basically all the different entities and how they relate to each other so in the example you gave you say this is an organization an organization has many members and these are the different

fields on the organization's and the members have and then depending on what screen I'm building what particular user interface I want to build I can specify the fields that I want and I can seamlessly traverse across these relationships so that I can get whatever data I want back in just a single response so so it's effectively putting a query language at the API endpoint itself so rather than having a database that's within the company you know siloed closed off and then you need to build the the API endpoints for each of the queries that you think your users

are going to be using it's effectively putting the query language right at the at the edge where the user can access it directly and make those complex curators in a very seamless kind of way yeah that's that's exactly right and you know there are a few you know restrictions on what you can do so it's not as generally powerful as clear language like sequel but it actually Maps really well to what you need if you're building an application so you know one of these limitations is that you basically need to know how all of this data is structured ahead of time so you have to define that schema and if you have data that you want to be able to query you can only query according to that schema

but you know that actually is how most applications are built so you generally know how your different enemy types relate to each other ahead of time and you can for example denormalize your data to make you know aggregations and various computations that you want available ahead of time you can also parameter rise these things soso graph QL allows you to for example if your wants a query a collection you can include parameters so you can do for example pagination or filtering or any of these kinds of things but essentially what you get back is the JSON object

that exactly matches the structure of those entities and peels that you requested this episode of epicenter is brought to you by cosmos the Internet of blockchains cosmos is live and we couldn't be more excited to see so many projects already building on it blockchain technologies are evolving fast and development shouldn't be one size fits all that's adapt developer you need the tools that will allow your dad to scale grow and evolve over time the Kosmos sdk is a user-friendly modular framework which allows you to customize or adapt to best suit your needs it's powered by tenorman core an advanced implementation of the BFT proof of state protocol Kosmos takes care of

networking and consensus and allows you to focus on building your application in your language of choice if the Ihram smart contracts will be supported soon and the sdk makes it simple for you to connect to other blockchains in the cosmos network if you have an idea for a DAP and would like to learn more about the cosmos SDK or if you'd like to connect your existing adapter cosmos visit cosmos network slash epicenter for represent our listeners the Kosmos team will reach out to answer your questions and help you get started we'd like to thank cosmos for those supportive epicenter so there's a nice blog post and the link to it in the show notes to where your co-founder talked a lot about you know kind of the benefit of graph QL and one of the ways he explained it was that with sequel based

databases you have this over fetching and under fetching problem can you explain what that is and how graph QL solves that yeah so this is a problem with most REST API endpoints so typically with rest if you request a resource you get back all of the fields that the server engineer thinks you might want for that resource and so it might be that I want to render a card that just has a user's name and profile photo but if I request the user object I actually get back you know 100 different fields everything that the server knows about that user and so that's not very efficient that the converse is maybe I

want the user's name and photo and also the count of their friends but you know the only way that I can get back the friend information is if I you know actually request all of their friend objects and so now you're also sending me back you know one friend for a you know object for each of those friends and so so that these are the types of problems that you end up with with the REST API is all the time and so you know graph QL solves that very elegantly by just sending you back the data that you're requesting no more no less okay and so then this is something that Facebook place if you build because okay of course they have so many queries

internally there was just okay we need to like safe cost safe bandwidth make it more efficient and build this kind of carry language because of that that's right the the other thing that Facebook deals with is you know with mobile apps people tend to run old versions of these applications and so they're supporting you know probably hundreds of different client versions if not more and so that makes it really difficult if you have this tight coupling between the user interface and the and the API because you know the backwards compatibility becomes a nightmare to support and this is actually quite similar to you know the situation that

we have in web 3 where you have a protocol that is going to be supporting a large number of clients you know one of the big benefits of web 3 is you can have a lot of different applications built on top of you know these protocols and so to be able to support you know multiple applications and multiple versions of those applications you end up having the same exact kind of situation so in one of the blog posts that you've published you mentioned that graph QL will likely be the preferred query language and database of the decentralized web what did you mean by

that and why do you think that is the case grab QL has been rising in popularity just in traditional web development so there's a lot of companies like you know github and Twitter and Yelp and and many others that you know I've already switched to graph QL and so this is already like a really big trend happening in web development generally and really with what you need when you're building applications is essentially like a standard for how you want to access your data and you basically need an abstraction and and you know we believe

that graph QL basically is just kind of the right level for this sort of abstraction where you don't want to have to know you know what blocks some data was you know updated in and all of the like internals of a blockchain if trying to build an application and so having an abstraction on top that is just the data model is we think you know the right level for how people who choose to build their apps and then you can kind of you know deal with all of the implementation details on the back side of that graph ul API

alright now switching over to dabs then so how do these problems relate to the dab space I mean because in the DAP space we don't have databases or SQL or any of that stuff we're just curing the blockchain right you know why would we need a bit database on top of the blockchain yeah so you know a lot of the applications that were built in the early days of a theorem were very simplistic but but you're exactly right that basically this indexing and query layer was completely missing from blockchains and you know I think it's just a function of you know these block chains have a lot of work to do it's a lot of work just to maintain consensus and so

you know in theory enforce focused on these like scalability problems and how to make it so you can build these smart contracts but indexing is is very much its its own kind of layer in the stack and it's its own problem altogether and you know I remember when you know people started building gaps on top of aetherium you know maybe there's only ten transactions that have gone into the smart contract and so you want to view you know you know your whatever your application is and it's it's easy to you know just show ten things on a screen and feel like everything's okay but but if you remember you know the

applications that started taking off suddenly you know there's like a hundred things on the screen a thousand things on the screen and suddenly you just have this really long list and you're scrolling really far and you know anyone who's used like a good Weber mobile app would would never think to themselves that that's an acceptable experience but you know we're you know we've been kind of building toys and it's kind of this growing up process to go from having like a proof of concept to having an application that people are happy to use and I think in that transition a lot of people have realized that like wait a minute you actually need to be able to you know search for specific data and filter and sort and paginate and these are all things that

you need to have indexes to enable and so one company that I think has a you know quite significant in the theory space is in furor right which is a consensus project that serves the tons of API calls about you know is that basically the problem that they aim to solve as well no so in fury doesn't do any indexing in fury' is just kind of a managed etherium node service so it is nice to not have to sync a whole node in order to start interfacing with etherium and so that's the problem that in pure resolves but then the indexing problem actually happens a level on top and so assuming that you can talk to in a

theory of node I you you basically have access to the JSON RPC interface that's that's the the interface that the interior nodes expose and this JSON RPC interface allows you to get you know certain fields that are stored in storage and your smart contracts but you're very limited in how you can access that data and so for example if I've got some some field that's you know a count that's stored in my smart contract that I can get that count but but if if I have like a list that's for example let's say that I'm a marketplace and I have a bunch of different listings

of things that people can buy and sell in the marketplace and and I want to filter to find a specific category of listings and then I want to sort that to just get you know the most expensive items for sale in that category for that I need to be able to do filtering and that's not something that is exposed in that JSON RPC interface and it's not exposed as it would be too expensive to actually expose that without having indices and and these the theory of nodes don't maintain these indices and and and so that's that missing indexing layer so Howard daps you know in companies

building two apps and all these projects that need index saying solving this problem now because a lot of apps are relying on indexing of data in one form or another and aetherium is fully decentralized so there must be a way that we're dealing with this issue yeah so every project that has gotten to the point where they're trying to build a really good web or mobile application has hit this issue and what most of them have done is build these custom proprietary indexing servers so you know they're they're like you know crap we can't actually you know run the queries that we need and so let's let's build a

server that you know this syncs data from from from etherium stuffs it into a sequel database you know spin serves it over an API and then our our front ends will just hit that custom API I'd say that that's probably what like 90 plus percent of the projects have done so far the other option is you if you want to keep your app completely decentralized then you can try to just sync everything on the client so basically you know in that case where you want to let people filter particular listings you could load up all of the listings on the client and then filter it locally on the client and there are a lot of

applications that do that also so you know that second option only works when you have some small amount of data or if you're willing to make the users wait a very very long time before you can show them a screen so some applications have chosen to do that but you know the alternative is you have to run and operate your own servers and you know it think it's it's one of these cognitive dissonance things where we're trying to build apps and a big part of building adapt is this idea of you know it's completely cerberus and you don't have to trust and you want to operate servers and infrastructure and yet in order to build

applications that are actually usable we have to do exactly that okay so let me get this straight just so we're clear here so currently what you're saying is that 90% of aetherium taps are writing their own proprietary software that's sitting on top of the etherium node that's being hosted on server infrastructure and they're serving SQL queries and through ap is allowing users that are using those tabs on their clients the query the databases of these centralized sort of choke points before they can even access the blockchain database that's exactly right very well said ok great I'm glad we got that settled

no I'm being a little facetious here because this is something that I've I've kind of stumbled upon recently I mean I hadn't really realized to what extent this was actually happening and through through digging and it's speaking to a lot of that developers about both you know different DevOps problems that they were having realized that yeah this is a big issue and one of the ways that people have talked to me about potentially solving this is making this proprietary layer open source and so then the idea is that you know different people out of the goodness of their hearts would host this infrastructure for example you have you know DAP a and

tap a has all this proprietary infrastructure they open source that's stuff and then users of those taps and maybe those who are particularly interested in the making sure that their dad client runs well and maybe they need to access to certain types of data like for example if you're running a fund on one of these taps might host that infrastructure I just thought well you know if you don't have the incentive mechanisms there to make that work well it's unlikely that we'll ever get a we'll get past this point of like hyper centralization yeah you're exactly right right we want um you know web 3 to operate on top of this like public global infrastructure and if we want that infrastructure to be sustainable then kind of payment needs to be built in there needs to be

incentives for operating the infrastructure and that's the only way to ensure that it continues to be available so my next question and I guess to me yeah maybe I'm missing something about the the the the technical infrastructure of aetherium but why wouldn't we just have this query language built into aetherium why would the theorem nodes themselves simply not expose graph QL endpoints so that the clients would have direct access to the node and direct access to these through these graph QL queries so so there actually is an effort to introduce graph QL natively into the nodes is an improvement for JSON RPC because the

theory of nodes actually do maintain a few indexes so for example if I want to get the ether balance for a particular account that's pretty fast because that is something that's indexed internally so that is something that we're going to be seeing but the the reason why the etherium nodes don't maintain more indices is that maintaining indexes is actually really expensive and which indexes to maintain as a function of which applications are getting built on top so you know with traditional sequel databases you should usually have like a DBA for example that's looking at like what are the slow queries you know let's add you know these specific indexes to make these queries faster and you know

there's no way for the etherium nodes to know which indices are the right ones to maintain and there's no incentive for them to do that so you know generally with software we look to build things in stacks right where there's like layers that build on top of each other and you know I think it really makes sense to separate out basically layer one from these kinds of you need to call this almost like a layer 2 problem right layer 1 is really concerned with things like consensus and you know data availability and and then I you know what they produce is flocks in a single you know global you

know state that everyone can agree on and then the problem of how do you organize that data so that it's easy to access for all the different applications that want to build on top really sits very cleanly as a separate layer on top in the stack cool well let's let's dive in a little bit in you know the graph I mean we've spoken about the problem of databases in query in traditional web application in the decentralized context you spoke a little bit about graph QL so what is it that the graph brings to the table so the graph is an indexing and query protocol for block chains and

storage networks so we index data from these different web three data sources and make it available over graph QL so the very first thing that we set out to do was to basically build a standardized way of doing this indexing so we've already kind of talked about how you know most of the projects in the space have done their own custom proprietary indexing servers and you know the first step towards you know being able to kind of introduce this indexing layer in the stack is to basically come up with a standardized way of doing that indexing

so we launched our standalone graph node July of last year and open sourced it and basically that defined the developer API for building what we call a sub graph and a sub graph basically defines how to do this indexing work in a way that can eventually run on a decentralized network and essentially what you do is you define here the data sources that I want to listen to here's a mapping script so it's a Turing complete language you know a way that you can transform that data at ingestion time and here's the graph trail schema for how I want to

be able to query that data and with that sub graph definition you now have a standardized way of indexing that data and making it available over Graziella basically the way it was before right is people built their own sequel databases and their own way of clearing that database and now you guys say ok there is basically a standard way of doing it you can use this graph the graph node and then use these sub graphs which is basically kind of like okay I want to have this data as long as I compile a to this format I can sort of add it and and it's thinning the nth idea we will have a graph node that contains all of the

different sub graphs or would I have like maybe my local node we have different sub graphs then you do well eventually the goal is to combine all of these graph nodes together into a global decentralized network so any graph node itself can choose which sub graphs it wants to index and what we want to do is actually open it up to a global marketplace and and and use market forces to do the resource allocation for which nodes are indexing what data so yeah that's let's get to the the public network essentials networking a

little bit but first or right now you guys are running this as basically like a hosted service why did you decide to you know start with that so you know we really believe in just shipping early and often and the best way to build software is to get it in people's hands and to work closely with users in developing that software and so it was really just kind of a pragmatic choice for basically coming up with what what are these intermediate milestones that we can hit where we can you know ship something make sure that we're solving real problems for developers and then improve it over time so the the first milestone that we had was just open sourcing the

graph node and we learned a ton after that milestone because projects were able to build on us and-and-and and so we were able to like quickly approve the software at that stage but you know it was still kind of there was this barrier to entry where people would have to run their own nodes if they wanted to use it and so the second milestone for us was launching the hosted service where we could run a bunch of nodes for you and you could just deploy to our nodes and we have a really nice user interface where you can see the status of as the nodes are like indexing these sub graphs and you can easily run queries in the browser so you can like test as you're developing and and that's something that

we launched a graph day genu area this year and and the next milestone for us is this the hybrid network and so it was really just kind of a practical intermediate step for making it really easy for people to build on the graph today right so basically the choice that that would have been is today they can either you know build their own sequel database and they hosted on their own server they built their own like querying an API for that or they do these sub graphs which is a little bit like developing their API sort of right they they define the data format and stuff and then you you guys were hosted

and they can just clearly that and you basically offer it as a hosted service exactly and and really you know we've been very kind of transparent about our goals to build this decentralized network from the very beginning so our model for the hosted service is we are going to be releasing a page here and the idea is to essentially just kind of cover the costs of running the infrastructure because you know as I mentioned it's really important that you know payment is built in and that this infrastructure is sustainable and so you know this is kind of one step along you know the path of you know

watching the decentralized network but you know this way it's really easy to kind of get started and we can start proving out a lot of you know these pieces that need to be built on top and so what about the hybrid Network what's that going to look like so the the hybrid network is that our next milestone where we no longer are running all of these indexing notes so I in the hybrid network we're going to have we're going to introduce our work token so we released at graft a Brandon Ramirez gave

a research talk where he described a bunch of new details about our decentralized network design and we published the specs for the hybrid network so you can check that out if you go to github.com slash graphic all slash research the token has two uses in the network one is a work token for people that want to run these indexing nodes and the second is for data curation and staking on the sub graphs themselves so for the work token model anyone can come in and you know stake tokens to run a graph node and then they can charge fees per query and that's

done in an open marketplace where they can set their own prices per query and so in the hybrid network we're going to open it up so that other people can run these notes but we're going to be running a centralized fisherman service and graph protocol the company is going to be very involved in kind of enforcing security in the network and it's kind of like intermediate phase so a couple things here and I guess we're get rid of Rick regarding scaling how does the scale because it occurs to me that so one you know the theory blockchain is already quite large in several hundred gigabytes now if you have to build an indexing service that has all of this

data but then structured in a way that presumably makes it much larger how do you deal with that but then also the other question is you know you mentioned earlier that with graph QL you have the schemas and it's unclear to me how we're gonna come up with like a general generally agreed upon schema that everybody agrees like this is a schema that a theory I'd have and that is optimized for all of the different taps right it's like one yeah unless I'm missing something here but one tap might require a different data schema than another tap yeah great great question so it's the first one on the scaling part you're exactly right

um from a sheer amount of data perspective the graph is probably going to be like one of the largest networks in terms of just how much data we need to you know store and make available and and and that's really kind of what our layer in the stack is focused on and so for example you know that's why we kind of have sharding which right now is kind of at the subgraph level and then it's actually going to be down to the individual index level each craft node only needs to index some very small subset of the data but yeah to make data you know very quickly accessible to clients all over the world you would really need to scale that out to to many many nodes that are geographically you

know distributed at the edge and and that's a core part of what we need to do now as far as coming up with these uh you know schemas that's where governance comes in so right now anyone can define their own schemas for their particular application for their stuff graphs and so you can think of sub graphs as being like kind of a unit of governance but over time one of the problems that we would love to solve is to you know add this layer of governance for the sub graphs and for these you know data types to exactly solve the problem that you're describing which is you know if if I have like a

very custom application you know I should be able to define my own ski but not have to talk to anybody but one of the big benefits of web 3 is to enable interoperability and if you want to you know provide these api's and you want to make data available so that many different applications can be built on top then you do need to have some coordination and so you know we've seen like standards bodies in the past that try to come up with you know standardized you know data formats and the indie standards bodies tends to move really slow and you know this is something that really excites me about etherium that we actually now have like a platform that enables kind of large-scale coordination and my hope is

that we can build great governance systems on top where for example people can you know vote on changes that they want to make to these globally shared schemas and then this can create a scalable way of evolving standards that enables for a lot more interoperability so so you mentioned that you guys let me sharding and that there you know there could be initially some sort of chart that you have you know per sub graph you know but boy what's the timeline here I mean sharding it seems to be not super close right if you look at all of the charting efforts like when do you think

like yeah what's the sort of timeline for like the hybrid network and that's gonna launch and and like a shard a network well luckily the problem is is quite a bit easier for us and it is for these layer 1 blockades um so you know layer ones are responsible for solving things like data availability and you know a whole set of issues that we don't need to solve right because an hour layer of the stack we only accept these like you know strong layer ones as inputs as our data sources and so we get to assume that that data will always be available and all we're doing is processing that

data and indexing it to make it more easily accessible and so and our layer the stack all we're concerned about is kind of quality of service you know if for some reason you know an underlying shard kind of goes offline like you know we can always rebuild the index and so that makes sharding a lot easier for us than for some of these other ones yeah let me see if I understand that because you know I guess if let's say you look at the helium and the shard in there well what's what's hard is you have these different charts and then you have like they have to basically be able to communicate with each other that you have these smart contract calls across shards and you know it gets quite complicated but here it would be like

okay I want a node on the graph and I can just go and say okay I'm gonna go on the augur subgraph and I just take that you know that kind of schema and I had just have that data and I don't have to care about any other sub graph right they just have to be able to serve had that information is that roughly correct yeah so as long as you have access to the underlying data sources that you need to process your sub graph then you're good to go and you can build up your index and and you can start serving clients so another question that came up for me is that you know you even at correspond you never call so solid we've been doing some of the stuff right like

getting the data of the blockchain putting in the sequel database reading it out and you know let's say different projects will have their own you know maybe analytics or insightful things like proprietary stuff that they're doing and that they may not want to expose exactly what they're querying because you know I could imagine if you can like see all of the queries that coinbase is doing like maybe you can figure out what they're gonna develop next or so how does privacy work in the graph Network yeah so I think there's a lot of use cases like you're describing and people are doing prop trading things like that where they don't want to make that data available

to others and for that you wouldn't need to use the graph it might be that you actually still want to build on the graph because it just makes it easier for you to use those tools internally but then you'll you'll just use it offline you can run your own graph node it's all open source and so that option is available to you you know the network that we're building is is all around public data and you know we think that there's going to be more and more and you know many orders of magnitude more open data that's going to become available as part of this move to block chains and so there if you have data that you want to make available you know for other developers and for different

applications that's when when you would use the graph but yeah actually I think that's a good point right because you could just like run you own the graph node with this necessary sub graphs and then query that and then that's not visible to the network right right yeah so so you have that option available just like you can run your own private etherium you know network yourself if you want to do that so moving now to economics we talked about the hybrid network a little bit so talk about some of the you know game theory and economics that go into the the hybrid networks which which is sort of the next

phase in the life of the product yeah so you know the the first step is making sure that you've got indexing nodes that are incentivized to index datasets so you know the the main incentive for the nodes is that they can set their own prices per query and so you know they can set those prices based on essentially demand so if there's a sub graph that isn't being used a lot and they need to cover their costs are doing the indexing and they could set the price is higher if you know it's a really profitable sub graph people are

clearing it all the time that they can set the price lower and and that's essentially the incentive then you want to have you know an incentive for people to to add data onto the graph and so we've got this kind of curation which is basically you can stake on a a sub graph and that entitles you to future revenue that's proportional to the number of queries on that stuff graph now there's actually a you know a lot of things that could make doing that quite difficult and this is

you know one element of the game theory that gets you know quite interesting so you would want to make it so that if I you know create a valuable data set then I can make more money and and that creates an incentive for people to add more and more valuable data onto the graph but but that can actually be quite easy to gain and so you know for example you know maybe I could find a sub graph that's doing really well and I could copy it and then try to get people to use my sub graph instead of yours I could choose to run my graph node and then query my own query myself where I'm just paying myself so I'm just moving

money from my left hand to my right hand to make it look like there's a lot of demand for this data set when really I'm just paying myself so in order to you know actually make it so that you can create like a long-term incentive for adding valuable data to the graph we have to like you know solve these kinds of problems yeah I know that nevermind to me though they're like ocean protocols you know spend some time looking at because they have a lot of the same same problems and everything pretty pretty tricky to get the game theory and incentives right there yeah now you know one thing that we're doing you know very different is that all of this is for public data and so we're not

trying to solve the problem you know someone's keeping this data private and you have to kind of like set up an agreement in order to access that data because we think that ultimately most information is going to be public information wants to be free and you know for the next like evolution of the web we should just assume that that's going to be the case but just because information is free and available it doesn't mean that you're not willing to pay for fast efficient secure access all right and so what you're paying for it isn't really for the information but it's for the work that somebody is doing to organize it

and make it easily accessible to you so you know one way of thinking about it is like you know maybe you burn so if I want to you know travel somewhere in San Francisco I can walk right that options available to me nothing stopping me from you know you know sign up my shoes and walking five miles but if I want a car to take me to my destination a lot faster that I can pay a little bit of money and you know get in a car and I can get there you know significantly faster and you know how fast I can get to my destination is basically a function of the number of cars that are

moving around the city so if uber only had 10 cars and you know it would take me longer to get a car and you know if if it were has you know 10 thousands or tens of thousands of cars in a city then the average time that I have to wait to get a car is going to be significantly shorter and so this is kind of you know a way to think about like you know liquidity in like you know an indexing marketplace where you know essentially you have this two-sided market where you know you have like the data producers and the data consumers and you want to have like a thriving marketplace where there's incentives for both sides to provide the service and to consume the

service so if that's not you point then to look at this as as a marketplace and it ties into my next question which is you know what are the what are the costs that we're looking at here so already as a DAP user their costs associated using the theory of network and making transactions and you know people are working on ways to improve things like like reducing the cost of maintaining state for instance but there's a bunch of costs there and then for DAP developers just costs as well whether those be development cost or cost to store data so there's already a bunch of cost here and then all know what with this marketplace that is like

a layer on top you're introducing another set of costs which is to access data like index sees and things like this I'm what what are your thoughts on you know the the increasing costs of using a blockchain and adding more layers of costs there yeah so you know block chains today are expensive but I would expect to see the costs go down significantly if you have computation that you need to have replicated on tens of thousands of machines that's going to be expensive and if you have data that you need to make available forever that's going to be expensive but you know I think you know we're already seeing a lot of you know new designs for

block chains that should be able to make it significantly cheaper and you know maybe different applications need different levels of security and and and so I think we'll see those cost goes down for the data layer we are introducing a cost but it's significantly cheaper so you know I would be surprised to see the costs end up somewhere around you know point zero zero zero one sense for clearly right like if you actually think about the cost of computation it's actually one of the cheapest things that a company you know spends money on so like in a traditional startup you know maybe

you're spending you know $1,000 $2,000 server costs at least you know in the early days whereas maybe you're spending you know over a hundred thousand dollars a year per engineer so you know I think actually the kind of social scalability costs I think are much more important to consider than the raw cost of computation and I think that when you actually build a payment into the protocols I think in the end we're going to see that you know it ends up being cheap enough when all is said and done so today you know most people are probably worth you know used to spending

$30.00 a month maybe on like actually it's probably more like $70 a month per like you know your mobile broadband your he's just spending on getting different services it's like your Netflix and and and whatever else it and I think that with web 3 you know people are it you know going to have to you know this is a kind of a new category it's a new expense for sure that people need to get used to like hey I'm now paying to access like web 3 but you know I think you know we can kind of look at how much that's going to cost in total and what users are gonna get back in return and I

think it's going to be more than worth it so basically have you know an explosion in the number of applications that you can use they all interoperate and I think that the kinds of apps that will be able to come out of that are going to be more than worth the costs so right now you guys have focused a money hiriam is it's the idea that this is gonna be you know the graph is gonna be his network that's gonna index you know all of the data or all of the block chains and make it clear well or are you so what's what's the plans there yes that's the vision yeah we're gonna index all the data and you know I think that

is going to become increasingly important as we see more and more block I already are seeing just with aetherium a lot of people are choosing to go to in fira for example because it's just an easier way to access data and imagine what's gonna happen when there's you know 10 or 20 different block changes and each one has many different charts and you have to figure out which full node to go talk to to get what data or your you know you yourself are gonna have to start running 50 full notes it's not really going to scale and so I think it's it's you know that that makes it even more important for there to be like a single network that's indexing all of the data across all of these different data sources and making it easily

available you know one thing that um you know I would want to kind of mention here is that we were essentially acting as a sort of aggregator web 3.0 ization and we've got you know these different networks and and you want everything to kind of be decentralized but you know from a user perspective there's a reason wife aggregators exist you know if you know the web itself was decentralized but like if I couldn't go to google.com to search for something and find a website you know those web pages wouldn't be very useful to me similarly

there's lots of different you know items that are for sale and different merchants on Amazon but if I couldn't browse and search for items for sale those merchants wouldn't be much much used to me and so you always have this kind of aggregation that becomes important to make things usable and typically you know those aggregators become your kind of centralized points of failure and that's exactly why decentralization is so important to us because you know we we want to make web 3 usable but we think it's really important that if you have this global

API that's indexing all of the world structured information and making available and that every application can use this global API to access its data and it's really important that that API is not owned and controlled by a single company or you know a single entity and and so you know that's why we're so passionate about making this a decentralized network so some projects are already using the graph so you recently I think in February had this event in San Francisco and seen several projects participated and spoke there and there you unveiled the the the

hosted version of the graph and the graph Explorer which at from a product perspective like I thought looked really nice I played around with the Explorer and it looks like you guys spent a lot of time on like design and making sure that it's quite usable and developer Docs and things like this so can you tell us a little bit about the projects that are using the graph now and maybe highlighting one that comes to mind and how they're using the product yes so we do have a bunch of projects using the graph already one that I'm really excited about is Maalik so if you go to Molech decom that's actually using the graph to power that user interface so sure many of your

listeners know moloch is a dao for funding yeoreum and infrastructure and it's got members and those members vote on different proposals for how they'd like to spend their funds and all of that is being indexed on the graph and empowering their user interface so you can go to the graph Explorer to find all of this data that's being indexed and is available for people to build applications and the graph comm slash Explorer and you can see a bunch of featured sub graphs there right now and then a bunch of community sub graphs so

it's really easy for anyone who wants to build a sub graph you deploy to the hostess and he'll show up in the Explorer and we just want to grow the amount of data that's being indexed and is available to people that want to build applications cool so let's let's talk about before we wrap up timeline and you know what's the road map so how long should be before like the hybrid network is really East and with some important points on the roadmap coming up yeah it's um we've got a bunch of features that we're going to be adding this year so one of the big ones is expanding to multiple blockchains we've got things like

pending transactions and something called the confirmations API we're really excited about so you know right now when you query the graph it just gives you back the latest state that it knows from whatever it theorem node is connected to but we think that one element that's been very overlooked in you know building adapts is communicating to users kind of the finality of various actions that that are taking place in the interface so you know a typical UI that you'd see today is you perform some action and then it sends you a link to ether scan where you

know you pop over to this other you know website to kind of see if this transaction has been accepted in a block and you know we've all kind of put up with this but I think it's a really great example of like a UX you know paradigm that is not going to scale to mainstream users and so being able to actually show in the interface like you know when you know a transactions been accepted when this actually has been performed you know this you know if I'm looking at a crypto collectible and it says that I own it you have I owned it for a while or has this just changed out of you know two blocks ago so so that's

the confirmations API coming later this year we've made a lot of progress on the hyper network so like I mentioned the specs are already available we'd love for people to jump in and provide feedback we've actually built the first version of the smart contracts to run that so we're doing a lot of like you know testing development internally I would expect it in the early part of next year but generally try not to comment on on dates too much but but that's that's kind of the roadmap and really what we're focused on this year is is really just getting adoption because we think the most important thing is to help daps build use usable

you know experiences as soon as possible and you know there's there something nice about this progression of actually starting centralized and decentralized over time is it it allows us to you know iterate a lot faster and you know if we get to a point where there's you know a hundred or more applications that are built on top and the graph works really well for them then we know that by the time we you know launch the decentralized network and it open it up to many more nodes to be doing the indexing that we have something that works really really well and even was very interesting to be speaking with you about the graph today and and great to

have you on to discuss this topic I think you know as I was alluding to earlier to by centralization in a theory 'm obviously isn't desirable and that's really what we the situation that we're in at the moment at least for most apps so the vision of the graph is one that i think will resonate with a lot of people and kind of goes in the direction that i think the original ideas behind aetherium meant to lead us towards one one i guess i wouldn't want to call it a criticism but one thing that we should be careful about is creating too much dependency on one service for so for

example if you know forever for any any additional layer that sits on top of aetherium i don't think it would be much better if you know every dab in the system would now depend on that one layer and should that layer cease to function or the economics are flawed or for some reason no longer really works well then you have a situation where it's not a whole lot better than in a centralized platform but hopefully the standard will allow for a multitude of marketplaces and similar types of products to emerge where people can choose you know which version of the graph through their pointing their

adapts tours yeah I think this idea of having it being open right which means open source so you can run your own notes and and you can verify everything yourself and and you know you can experiment and and and that nobody's kind of locked into any one particular solution is really important and and so we we agree with that that vision you know completely great thanks for coming on and we look forward to seeing future developments of the graph and so we will be linking to everything we've talked about the blog posts the website the Explorer the github and everything in in

the show notes is there anywhere else that you would like to point people if you want to get involved though other than those resources already mentioned yeah we're most active just on Twitter and medium great thanks again for going on thank you so much we release new episodes of epicentre every week click here to subscribe for hundreds of insightful interviews with some of the leading minds in blockchain and crypto you can also listen to the audio version of the podcast on iTunes Spotify stitcher and other podcast apps click here for a full list of places where you can listen thanks for watching at the center and we hope you'll join us for our next episode