Data Coprocessors For Sub-Second Blockchains.

Transcript

We have 1 minute. Hello. Hello. Welcome. I hope you guys have a good lunch and are eager to get into data core processors and indexing. So, my name is Pranay Valsan. I work at Covalent. So, we've been working in this space now for 8 years. And basically what we've been providing as a service is indexing, right?

What I want to do today is take you through the journey of what indexing looked like maybe 5 years before, what it looks like right now, and how these things are going to evolve with all the changes that are happening on the protocol layer. Yep. So, just by show of hands, just to get an understanding of the room, how many of you have actually integrated an indexer into an application that you've built? Okay. That tells me that I have to fill a lot of gaps.

How many of you think it's easy to build an indexer on Ethereum, for example? Zero again. Okay. And uh So, that tells me enough that I need to know to cover the gaps. So, what are we covering today? Like I said, the history of what blockchains were, and what the foundational data extraction, transform, and load stack looked like. The second part will cover what the use cases driving demand for block space are, and how block space has become a commodity. And finally, we'll be ending with the architecture of what is able to provision this with block data streaming that we've come up

with at Covalent. Let's get started. Um Yeah. So, historically speaking, when block times were a few seconds to a few minutes, uh blockchains were considered to be a settlement layer. And users worried about self-custody, where not your keys, not your crypto was the overarching theme, right? And what they cared about is that their funds settled, and the fact that they were on a network was decentralized enough so they could leave the system. So, the walk-away test, as we call it, right? If you [snorts] wanted to exit the system, you could. So, using that uh in mind, the builders emphasized on consensus mechanisms that were really

good. So, early on it was proof of stake, sorry, proof of work, and then we migrated to proof of stake. And they also focused the the developers also focused on the ability to audit contracts that were deployed on chain. So, contracts have become the bedrock of how transactions are actually processed in the ecosystem. And as long as there were enough nodes supporting your bytecode, you could say that your transactions would be executed, and that you would have a result from that execution. That was the old normal. And that allowed us to create a kind of generalized transaction engine that we could build for EVM.

Here, the idea was that you have this piece of software that sits in between the RPC nodes and the customers. And what the software does is it basically makes adjacent RPC calls to the underlying RPC node and fills a Postgres database. Now, this Postgres database has multiple tables. You can have tables for transactions, for logs, for traces, for anything that you cared about as a developer, you could put that in a database, and then expect that to be the source of truth.

And now that you have this Postgres database, which is a single source of truth, you can wrap it with any kind of library, maybe Django back end, or even Java APIs, which could read, update, delete any sort of value. Mostly read in this case, because once it's written, it's final, right? And so, this is easy to scale, and we could scale this for over 100 chains. Most of them were EVM chains, because the generalized transaction engine had the same JSON RPC interface. And so, that standard allows indexers to scale quite easily. But what we're starting to see now is with chains that are super fast, especially with like lower block times, say close to like a few hundred milliseconds, 200 milliseconds, or even

less, right? There are some chains that do a thousand transactions per second. Now, how do you index that in this world? That's completely not possible, because the size of the underlying tables keeps increasing. And the cost of storage rises asymptotically with the size of the database. So, the Postgres database here that I've posted is for three different chains, Base, Ethereum, and BSC mainnet. And just the logs table, the transactions table, and say the contract balances table is a few terabytes. Now, imagine doing this for 50 chains.

This easily goes to 100 terabytes. And imagine having to serve that on SSD based drive, where you have a piece of software running on top. It optimizes for query time, where you have to get the data in the shortest amount of time possible, but it also has to be accurate. Furthermore, if you have to do any kind of analytical queries, it becomes even more difficult, because now you have to aggregate amongst multiple tables, and you have to make the joins between tables by using foreign keys, right? So, in the world where chains are like moving super fast, and blocks are finalizing within 100, 200 milliseconds, this just simply stops stops working, right?

>> [snorts] >> Um so, initially, maybe about 3 years ago, before chains get came this fast, because the size of the database was growing, we thought one way to kind of mitigate that problem would be to distribute the workload. It's obvious. So, we came up with the notion of a distributed ETL pipeline using the Ethereum Wayback Machine. And the idea is pretty simple. You have a network of validators. Each validator is assigned a specific role in the pipeline, whether it's extraction, transformation, or load, right? And using crypto-economic incentives, using an underlying token, we could get them to coordinate on the different parts of the ETL pipeline.

But obviously, this has certain limitations, too, because the most important part of the pipeline is extraction, right? And so, all the assets converged towards the producers of the underlying artifact, and the network became a bit imbalanced. And so, the incentives were progressing towards concentration of capital towards the extraction part alone. And so, the other parts of the network, especially close to the application layer, for example, for the query node operators for the applications, >> [snorts] >> uh decrease. And so, we had to come up with an understanding of how to suppress this kind of artificial concentration of capital for validators.

And so, we have to again think about the token economics about how we distribute the token and the valuation for the different parts of this extraction pipeline, and uh basically come up with a verification mechanism, which is lightweight. So, in the in the current sort of conception of how blockchains have evolved, um because of the lower block times, what we see is the notion of the settlement layer has kind of shifted. It's gone from being a place where you would put a transaction and expect it to be completed to a world where it's a bulletin board, where a piece of information exists for a very short period of time, and you can verify that it's true, but nobody needs to keep a

hold of it. And there's this EIP called EIP 4444 that actually plans to expire all the historical state of Ethereum and move towards a more stateless world, where clients can specify what's the last block that they want to sync from, or what's what's the first block, and what's the last block that they want to sync from. >> [snorts] >> So, not all RPC nodes then have the historical state of the chain. So, this is the problem that we are facing now is with all these different changes, the users want faster trades, they want responsive balance updates on their applications, they still want the exact data, and they want to be able to access it at any given time. So, we have to

fulfill those requirements, while builders are constantly pushing on the number of transactions that they want to do by having bigger blob sizes, by decreasing the fees of the transactions, and having faster finality. >> [snorts] >> So, this world is already here. That's what I'm trying to say is we're doing 50,000 transactions per second on chains like Hyper Hyper Core and Solana and even Mega Eights. >> [snorts] >> And the primary receivers of this data has shifted. Early on, it was human beings who were looking at their transactions going like, "Okay, this takes about 12 seconds. I'll wait for it to complete." But now it's agents.

Agents don't require a human in the loop. They want to be able to make decisions on new data that's coming in, and they want to be able to make it fast. So, the latency requirements have drastically moved from like a few seconds to a few milliseconds. And imagine that within a blink of an eye, which is 150 milliseconds, 100 transactions have occurred on chain. So, it's literally not possible for a human being to catch up at this rate. Um and what this leads to is a difference between what's at the tip of the chain and what the agents know from RPC providers or from data indexers. And this gap is growing as the as as more blocks are being added, the

gap between the tip of the chain and what the agents can receive from the underlying indexer nodes is growing further and further. And this leads to what we call state blindness. The applications at the front essentially don't know what the actual status because they're so far back from what they can actually receive. >> [snorts] >> And uh this is the example that I wanted to post is we have to rethink the arch. Uh so Lefteris is the founder of Rotki app, which is a self-custodial um portfolio management app. And what he's trying to say here is the indexer situation for EVMs is extremely bad.

It's bad to the point that he's suggesting that it's not possible to use base and optimism because the indexers are not able to provide accurate data. So, if people are relying on this data, they make wrong assumptions uh and the accounting is wrong and the taxes are wrong. And so, he personally stopped using the chain. Right? So, you don't want to be in this world where you have super fast transactions and you have finality and security, but you don't actually know what's on chain. Um and this is where we have to move our thinking towards block stream processing and the medallion architecture. So, let's get into that.

The easy way to get out of this problem is to actually go back to the source. So, what's the source of the data? It's the actual RPC node, right? At the time that an RPC node is syncing, you can run an engine that exports the entire block along with the traces of the block in a protobuf binary format, right? And you can do this by scaling workers. You can have for example, if you wanted to index 10 million blocks, you can have 10 different workers at different block segments, say block zero to block 1 million, then one from 1 million to 2 million, etc. And now you can concurrently index 10 million blocks by these 10 different workers and then

linearize it at the time that it's required at the end of the pipeline and push it to a message event system. So, in this case we're using Kafka for the underlying block stream processing. >> [snorts] >> So, now that you can use this underlying block stream processing, you can scale it for multiple chains. So, you can have one chain produce into its own separate topic and you can have multiple chains doing this into the Kafka cluster. And then independently you can decide which part of that data is the lowest latency high throughput data that you might need. So, we get a lot of storage savings as a result of what we call tiered storage. The idea here is very similar to like expiring data, but still

keeping it in the sense that remote long-term storage holds data from like genesis to the last year and from the last year to like the month and last 2 weeks, you keep that on a hot SSD, right? So, you can still have super fast access rights and reads from the SSD, while if you want to make a point query to historical data, you can still use the long-term storage. >> [snorts] >> And this allows us to reduce the cost of of uh keeping this data for a long time >> [snorts] >> and also have multiple consumers that according to the needs of the consumers can point towards that part of the storage.

And [snorts] this unlocks a lot of more different applications. So, [snorts] now that you have um all the data in Kafka clusters which are partitioned by topics per chain, you can have different consumers consume it from different offsets. So, depending on how much data you need to do your analysis, whether it's just the 5 minutes of data from from this day or maybe 1 year's worth of data, you can point the offset of your consumer, right? To that particular point and start consuming from there. And this essentially allows you to also have connectors for clients where a connector is something that consumes from a given topic and here the topic can be um say transfers for a given chain and the sync can be like another user queue

or it can be their own database or it can be their own data warehouse. So, this allows us to like decouple the production of the data from the consume consumption of the data. And then the consumers can define what kind of a format they need their data in instead of it having to be a Postgres database where the underlying schema is predefined by us. This allows us a lot of convenience in spinning up jobs ephemerally, giving you the data you want and then delivering the results. >> [snorts] >> Right. Um and this decoupling of producer-consumer architectures allows you to have this notion of a medallion architecture where the medallion essentially is divided as bronze, silver, and gold, where bronze is that

raw block information with all the traces underneath. And then you have a process in the middle that takes it reads from this topic and produces data into a different topic where it can have enrichments. These enrichments can be ABI decoding, token metadata, some sort of annotation like USD pricing of the token, etc. And this allows you to compress the information from just raw data, which is like for base it's around 90 terabytes, to for example transfers, which is only 1.2 terabytes. So, you already get about 10x savings in terms of space, right? And then you can take [snorts] the silver data and further refine it where you have another process that can extract from transfers what are the DEX swaps that are occurring,

>> [snorts] >> what are the OHLCV candles, so open high low close, the different volumes that you care about. So, you can derive actionable information from the gold data. And then you can point your consumer to whichever piece of information you need at the lowest latency with the highest throughput. So, here I have some real-world data compression use cases where we've gone from bronze data, which is a few terabytes, to actual actionable data um in a few gigabytes, right? And we can do this pretty quickly. So, I I have a table here on the right which has live metrics from this architecture where end-to-end latency going from bronze data to uh gold data is a few seconds, while the actual

processing time of the block is a few milliseconds. So, this gives us both high storage savings as well as time savings. Instead of waiting for the indexer to index everything completely, you can just choose what amount of data you want and just focus on it using this producer-consumer decoupling. And the primary use cases for this data that we're seeing is in uh DeFi where you have these real-time agents who are processing OHLCV metrics, are doing DEX arbitrage, or doing some sort of liquidation monitoring.

And the other examples here are also stablecoin monitoring and actual analytics where you want to aggregate a lot of data and derive some insights from it. So, yeah. Um that's basically the presentation and here I have a single slide that kind of sums up um everything that I talked about, which is we as a community are facing the challenge of state blindness as a result of block space becoming a commodity. And uh sub-second blockchains exist. Um block blocks finalize in a few milliseconds, but traditional indexers are not able to keep up. And so, people are stopping to use applications on those chains. So, our solution to that is to have this Kafka block stream basic

engine built in along with the node where you can differentiate between bronze, silver, and gold data and you can define what the schema of the underlying data looks like based on your consumption requirements instead of what the the producer wants, right? And the outcome is that using this architecture we've been able to scale to up to eight different chains uh with up to 1,000 transactions per second that we're processing with end-to-end latencies of less than 2 seconds, getting us 30x data compression, and this being essentially agent ready.

Yep. That was my talk. Thank you so much. Any questions? Thank you for presentation. Anyone have a question? No? Yes. >> [music] >> Um yeah. Um we observe this um you know, scale of um very fast throughput and increase in TPS, but yeah. Isn't it sometimes at the cost of um finality? You know, transaction finality can be more than you know, for example, on Solana and other chain like that.

Yes, but the question you have to ask is um do the agents care about finality as well or do you want to have an action taken by them uh in that time that is available? Yeah. So, there's this notion of pre-confirmed blocks. And of course, blocks take 2 seconds on base, but you can have pre-confirmed blocks which are optimistically uh proven to be enough um for a confirmed block. And you can actually act within that time. If you're for example, making an MEV bot or an arbitrage bot, that difference of I would say like 100 uh 1,200 milliseconds, 1,800 milliseconds is is a lot. And it in the in that kind of

scenario, you have no other choice but to kind of default to the fastest um um lowest latency data that you can get. Of course, if you want to aggregate and actually make a decision, then you're already late by the time you you do it. And so, there is this trade-off between accuracy and latency um that you have to deal with. Even as an indexer, we can provide you bronze data within 100 milliseconds. But if you want well-defined transformed data that you can act on, that we are sure about and the ecosystem is sure about, then it will take you 1 to 2 seconds. So we haven't sort of solved that final, you know, last mile problem just yet.

Another question? Okay. Question regarding transforming maybe the the latency issue from trade five topics partially have trading in in milliseconds like 10 to 20 milliseconds. Yes. Do you think you will be heading to that goal also? So could you provide that finality in near future also in 10 to 20 milliseconds?

Yes and no. So yes because you can run a custom indexing engine on the RPC node that's closest co-located with the server that might be doing trade five. Right? So if your server for example located in Japan, we can co-locate the data co-processor along with the RPC node there to decrease the ping times. That could be one way. The other way could be probably running more concurrent processes where they're distributed across the world and the one that catches the data first is the one that sends it to you.

Okay thanks. It was last last question. Thank you. >> Okay. Thank you so much. >> [music] >> Thank [music] you. >> [music]