DAPPCON 2024: Modern Blockchain Indexing: Syncing Millions Of Events In Seconds - Denham Preen

Transcript

hello fit no I'm just kidding I'm from Cape Town um by a show of hands who here has written a single line of code in their career all right and who's a a day-to-day developer they would call themselves a developer all right cool just getting an idea um today I'm going to talk about modern blockchain indexing and what that looks like so I'm going to start with just this teaser slide and we can we'll come back to it we'll unpack a little bit more but it's just to kind of highlight that when I talk about modern indexing we working in a space where there's a lot more data than it used to be when you know blockchains started and

in order to index all this data we need to apply some web 2 Technologies into the web 3 space to really make this faster but come back to the slide I'll I'll touch on it a little bit more first a little bit about myself uh my name is denim like the genes uh I'm the co-founder of nvo uh I'm a Believer in the tech and not the hype um my academic backgrounds in computer science and economics uh I'm an anarchist at heart I can see the Emojis are got a bit wonk and I've been developing in the space since main net gas prices weren't so high uh a little bit more about myself and the ecosystem I'm a py holder uh recently a grounds St member very cool D

if you want to check it out it's a noun St Fork about coffee and I think most notably I'm a safeg guardian cool buy builders for Builders now this is going to be maybe a little bit of a a brag but I just want to maybe add some credibility to uh who I am and the team I work with but we are big Builders of heart we've attended several hackathons around the world uh not the first one in Cape Town uh going to Denver um Bangalore amongst a couple others we've been fortunate enough to present at events in Davos New York uh Barcelona amongst a few others too and we some of the uh smart contracts we've

architectured have uh securely for several years held over $75 million in value um excuse that slight brag all right so uh my talk on Modern blockchain indexing begins with uh a story so there were three little Builders founding a def a Chad defi startup and after a year in production their data indexing would take a week to deploy uh at the time we were using subgraphs and it was quite harrowing or was quite scary to think that if our indexer went down and we had to start it from fresh it would take a week for our kind of backend system to present data

to the front end for it to deploy which is quite a long time and this is quite scary in the concept of going down and we actually got a grant from the graph to to build like a more robust version it was called subgraph from crashable and eventually we uh decided to take the leap and start uh building an index set for ourselves all right and this indexer for ourselves we felt it was materially better and so what we started to do was generalize it so that other applications can use it um cool coming back to the slide and we can we can unpack it for a little bit uh this is the amount of time it takes to index a single Unis swap uh pool and keep in

mind there are thousands and thousands of Unis swap pools uh running benchmarks across a few different uh indexes you can see with the graph on the one end uh subgraphs it's just under 17 hours and with us it's just under 10 minutes so that's quite a noticeable uh jump and what that practically looks like um in the context of float that's about 10 14x uh speed Improvement and so it took our one week deployment time into 1.6 hours which is quite substantial and allows for much faster um iteration cycle uh just think about you know when a product team comes

to you and says cool we want this feature and it's like cool I started work on it it's done it takes a week to deploy and they say okay let's change it up a little bit and you got to redeploy it then you have to tell them oh it's going to take another week and this is quite a narrow example um In Like A A Defined case where it was about a week but for example the unop indexer takes about V2 takes about 40 days to deploy so that's quite quite a substantial difference all right build bigger ship faster uh we have some of these around but I just use that to kind of break the flow so at this point we've we've ched about why we need it why we needed it and we've touched on how much faster it is but I

think if we want to uh chat about modern blockchain inting we should dive into how we actually do it and what's what that difference looks like so we let's jump back a little bit let's take a look at how you get blockchain data all right so you get blockchain data from a node and you usually make an IPC request uh a remote procedure call but essentially you say cool node this is the function I want to call fetch that data for me and send it back and you can see in this diagram we've got uh the client and the blockchain node and we've got two kind of example IPC requests there I think it's meant to be get block and not get block data and the second one is get logs but uh essentially what's happening is you send a request You To Tot node

what data you want to get and it sends it back um to the client now this has two major bottlenecks so we identified two major bottlenecks in this kind of process uh oh and sorry one other thing I just want to chat about as well uh maybe I'll get to it okay so the two the two challenges were Network and bandwidth I.E you sending uh data over IPC and it's it's a lot of data you're collecting so it's nly slow you limit it by your internet speed um the second thing which is a little bit more vague and open is application design so a blockchain node is designed for consensus it's designed for uh kind of adding new blocks to the chain it's not

designed for getting data however it can be used for getting data it it isn't designed specifically to to be performant in getting data now something like get logs is a is a method you use to get event data when people interact with smart contracts and that can build up over time you can get quite a lot of events so it can be quite a lot of data to transfer over and so we identified this this kind of challenge sorry just to give you like maybe a very very crude example of what this uh could look like and don't shoot me but essentially imagine you have three gigabyt of data you want to like collect and you're storing it in a text File versus a database it's obviously going to be faster to fetch from a

database now that's very crude but I just use it for elative purposes to touch on uh that nodes aren't designed for getting data so pretty straightforward we developed hypersync and hypersync in the simplest terms is a node designed for reading data all right so it's designed for quering really fast um we say mega fast Mega flexible and real time which is which is awesome and what we did to kind of change things up and a little bit more of the technical behind the scenes we developed this this note so it doesn't perform consensus it's more of just a a data leg so it won't keep your blockchain alive but it

will be um it's kind of focused on getting data and we we really just applied a lot of uh web 2 Technologies to the web 3 space so we took modern uh database Technologies and applied it into the web 3 space so here's kind of a simple example uh of something we implemented which is called par and the way we store the data so you might be familiar with kind of Rob based records we will store data in rows and this we shift it and we store in columns seems pretty simple but interestingly enough you reading this data from disk and so to search through disk it's a lot faster if the indexes are next to each other rather than in a

row based format super simple but really has some kind of notable improvements on how you get that data the alternative would be if you could get everything into RAM but if you look at something like uh arbitrum which is 7 terab for full iov node you obviously can't store 7 terabytes in Ram so you've got to start optimizing for for disc reading um cool another thing any rust stations out there any developers in Rust um it's a extremely high performant language uh that does some clever things with me memory and we built the system in this uh this diagram I'm showing you here is actually just the comp

comparisons between different uh clients uh different nodes and you can see how much WTH which is a paradigm recent implementation is quite streaks ahead you can see time to sync 50 hours versus the next up which is 5 days and G and nethermind getting to one month so quite a quite a jump in performance another bit of the the secret Source just to touch on on hypersync is we implemented some called iio uring I won't dive into it too much but uh with a package called glomo a rust crate we were able to optimize read and write to disk so again some super powerful technology but

completely available and accessible um and it allowed us to increase the read and write speed to disk um cool so a lot of people think of indexing as uh simil SAR to subgraphs where you have an application you've got events coming out of your contract and you want to understand how you know your users inter actors and you aggregate that data now that in this context is hyper index and hypersync what I've been chatting about now is a kind of full represent representation of the blockchain and so it's much more raw data but it's the engine that plugs into hyper index and allows us to index extremely fast so that diagram I was

showing you earlier that was index you know specific uh events from a contract from a Unis swap swap pool and that's on hyper index and beneath the hood is this engine hypers sync and that's what we use to to uh facilitate it to be so fast and be a lot more developer friendly um I also want to just give a special mention to hyper RPC so hyper sync is our own custom API and it's because RPC is UN for bottleneck by uh Network and bandwidth and so what we did to make a little bit more plug andplay a little more friendly to people trying to access data is we put a RPC layer or

kind of conversion on top of hypersync and so you can make normal RPC queries and that'll automatically get converted into hypers sync uh queries behind the scenes it works really nicely for plug and play but unfortunately again it's um it's still sending the data of RPC and and hence a little bit uh we don't really get the speed optimizations from Network and bandwidth however you can still search massive uh data sets and get uh the the queries you're trying to get a lot faster cool so defitely uh follow my co-founder John John he's uh done some really fantastic examples of hypers sync in action uh specifically like you know

aggregating or making queries into getting uh raw data particularly good for analytics other kind of applications you you want to look into and here he's got two examples the one is in under 15 minutes he was able to get 80 89 million transfer events on eth Manet for usdc that's pretty impressive that within a kind of single query you can wait a little bit and you'll have this this response uh the other example there is is something I think super cool is uh he he's essentially passed in an address and got all the transfers and and related to that address and that's in 0.67 seconds now with speed like that you could build a block Explorer with no backend you

simply get the data directly from the raw data directly from the chain or hypers sync in this instance so really cool to see these he's got a he's got 30 30 uh examples of this coming out and they're coming out day today which is pretty cool definitely recommend checking it out and then yeah we we uh have kind of gone to Market quite recently we quite an nent product we are getting a lot of really fantastic feedback which has been pretty exciting I think if you you kind of read into this it's like crazy how unly faster product is it really just highlights that some the the feedback we we've been getting is around speed and it's been it's been pretty motivating it's been very

exciting uh and then wrapping up uh we've got docs everything in N.D you can check it out um the link to Discord Twitter Etc will be there however that's our Twitter handle NV indexer and it's also our YouTube handle there's some really great uh speedruns kind of two minutes to to break down everything and then that's my Twitter handle below den and PR cool thanks [Applause] everyone ah thank are there any questions yeah we have time for a few questions so don't leave this stage yet um yeah does anyone want to start here's a

question thank you CLI just one second okay oh thank you great presentation I was just wondering about some details you were talking about index and events on uh Unis swap version three pools but it's kind of wake if you simply retrieve and are able to uh basically query all the swappy ones it's not that interesting but can you reconstruct the internal state of the given pool from the events in that same amount of time the like active TS and where the liquidity is located stuff like that uh I mean so the hyper index there is you you can kind of resolve the

events and you can do what you want with those events and you can aggregate that into the you basically get a huge data set right yeah so I suppose hypersync is kind of like the raw blockchain information hyper index is the layer above it in which you can you can choose how you want to aggregate your data into your database and you can uh but great question you can store it as you know like how many deposits the users made what the total value of the deposits is uh and you can get much richer data uh at that level it's a lot um kind of easier at hyper index but sometimes you might have a custom use case you need to do something specific and then you can use hypersync which is a bit more raw data but uh much more flexible mhm thank you

cool there's an thanks for the talk um is there a way to get um access to Raw data files because like you could analyze files on using AWS Athena or something absolutely yeah so hypers sync uh we've got three clients so we've got a python client we've got a r clients we go to JavaScript clients and if you use those hypers clients directly you can uh import the data you want into park files and then you can do a whole bunch of cool things like we've seen it uh plugged into click house um yeah a few other applications and then you can do your kind of data querying yeah cool anyone else we still have a

few minutes all right um okay then thank you so much denim for your great presentation and let's uh wait a few more minutes for the upcoming one cool thanks everyone