Instrumentation of Clients to Extract All Observable Blockchain Data - Matthieu Vachon - StreamingFa

Transcript

Hi everyone, I'm uh Matthew Vashon. I'm CTO at streamingf fast and today I'm going to present you our fires indexer uh stack especially in relation with it clients and how we leverage the get live tracing feature to implement it. Sorry, wrong thing. Um let's talk uh to start about our design principle that led us to the the the development of the solution. So the first very uh important thing that uh was in our mind when we started to develop the fires indexer was it to be streaming first. uh we think that having the data as fast as possible is the most important thing people needs um when they consume data and uh versus

for example a polling system based on the the existing RPC uh methodology. Uh second thing that uh we wanted to baked in in our system was to have a very deep tracing implementation uh of the data. Of course, we wanted the normal transaction calls, but we uh most importantly wanted to actually record all the state changes that happen as the node execute a transaction. So, the balance change, the storage change, the nonchange and stuff like that. Uh we think that uh those state changes are kind of underused right now because they are not broadly available for people to use. So, we wanted to have that in our um our indexer. And the last thing that uh was really important for us was that

the fires indexer needed to be blockchain agnostic. So we built it in a way that it was actually possible to index uh any blockchain as long as the node has some um instrumentation capabilities to output some data. Uh let's go with a quick overview of uh our get fires tracer evolution over the years. Uh we started our first version in 2020 uh where we actually kind of implemented a quite aish solution where as the node was executing transaction we simply emitted a bunch of print ends at the different places where uh state changes was happening as well as transaction start uh and block start and those kind of thing. Uh it was very uh

quick and dirty but it worked pretty well for a while. Uh but over the the next two years after that we actually started to uh kind of productionize our tracer and started to implement an interface uh for the tracing hooks uh at different places because we started to need to uh maintain uh more forks like BSC, Polygon and most of the EVM compatible chain that are using GT as their execution client. Um and uh finally uh in 2024 we actually started to pair with the GT team and Cena in particular to bring uh most of the live tracing feature that is known today in G. Uh namely we uh added uh with Cena a first version of uh block tracing. So

when the blockchain actually executes the transaction and synchronize with the network to actually be able to uh live trace what's happening. And we also added the missing state hooks that uh we required for balance storage and non changes and all those state changes that was happening. Uh last year in 2025 we actually start to be uh all our system in production are actually leveraging the get live tracing feature for all the network that we support. uh which means polygon, arbitrum, optimism, say coosmos uh those that are coosmos evm based that use get and uh we are running with the get live tracing node uh nowadays.

Let me explain you uh quickly uh what the firewalls indexer protocol looks like. Uh it's a very simple protocol that is uh composed of two main things. First, it's textbased and uh it uses protobuff data to exchange the information from the node to uh another uh reader part that I'm going to explain after. As you can see uh the the textbased protocol that we are using is is simply composed of two messages. uh the first one being an initialization message that simply kind of uh make the exchange with the reader about what version is going to be uh read later and uh after that we have the fire block

message which contains uh the block metadata. So block ash number parent and final number and uh a block protobuff model exchanged in B 64 over uh standard uh output. So it's really kind of simply a text uh version that is get uh getting output to the standard uh output of the node and the block protobuff that we have is simply as the live tracing is is running we simply accumulate all transaction trace and state changes into a full block model that uh is for Ethereum only and contains everything that we've traced so far and at the end it's emitted as we can see from a single Google message that is going to be later

read by some other system. This other systems is what we could call the files indexing stack and it's composed of four major components all working together. Uh the first component uh we call it the reader part and this responsibility is simply to spin up a get process and hook into its standard output to read what we just saw uh earlier about the protocol. And this reader node uh is going to be uh fed is going to fed the the next component in line which is what we call internally the merger and the merger responsibility is to transform what it read into a flat files storage system. So we um we merge uh 100 blocks together into a simple uh

filebased format binary this time uh containing uh simply a bunch of protobuff blocks all together in a sequence and this enables uh the persistence of the data like a bit like error files that exist on Ethereum today and this can then be uh saved to disk or sent to a cloud storage provider what we use in our production system. After that, uh the next uh component that is really important is the live source that we call the relayer internally. And this uh live source component is responsible of receiving the live data extracted from the node and to forward it to the other component that needs this uh livveness and um this is actually joined

with the historical blocks in some ways to create a unified stream of data from genesis to edlock. Uh the next two uh the next last two components are what we could call uh the the consumer product which are firewalls and substreams. Firewalls is a simple gRPC server that uh will uh simply send you the protobuff block that um we uh created and you will receive the full block with all the information. So no need to do another RPC call. You get the fully traced block with everything in it, all the transaction that has been executed, all the state changes with the pre the post data directly in there. And

finally, substreams, it's a um um let's say a transformation engine with fully parallel execution built in. So we chunkify the the the chain into segment of 1,000 blocks which are then executed um you send in fact a wasam binary to us and we executed for you in parallel over all the chain uh with different worker depending of what you provided as workers. So we can work for example uh 1,000 worker which would mean around 100 uh,000 uh blocks at a time.

uh what you get from uh those the stack of indexing. First you get a gRPC stream with the fully traced data. Uh this is a unified stream fully sequential that you will receive blocks one after the other with the fully uh traced data. The second thing that uh you get from there it's a forkaware uh stream of information. uh since the node is sending the block as its trace, we uh actually kind of created a component that we call the for DB which is responsible of tracking which block is the canonical one for the chain at end and we can also record um which block has been just undo by the

the new block and we send that over to the gRPC so that the the customer can actually act on the fork um signal and do whatever it needs to undo what just happened depending of his own uh data pipeline. And finally, one of the the last very great feature that you get with that system, you get a cursoring system. Uh so for each blog that we send you, we also send you a cursor that uniquely identify the blog that you just received. What it it gives you is that uh as soon as you disconnect and you reconnect, you simply send the last cursor that you process and we will resume the stream exactly where you left

off. This uh make sure that you will never miss any data because since you have a cursor, we can simply uh send you uh the data that you left your work in. And to finish, let's talk a little bit about uh what we are uh working on right now. So we are working on having uh flash blocks fully trace with the get live tracing. So we have a flash block prototype in G that is able to uh live trace using the live tracing system of get the flash block and emit it back in the gRPC stream directly. So um we are aiming to ship that in Q1 2026 and

hopefully we'll be able to uh upstream some of the work that we did uh into get or optimism get directly. Another thing that we're working on right now is uh Bezu instrumentation. So we uh are starting to work on porting our firewalls ethereum tracer to Bezu and uh again we hope that we are we we will be uh able to open some PR on Bezu to bring the live tracing feature on the Bezu and uh Ethereum client itself and finally we have also work on uh porting a bit of our fires tracer uh using red xxx uh feature uh it's a bit different than uh what we have in our get live tracing but

uh we leverage the red xx journal to actually extract the state changes that we need and we have a let's say a start of an alpha prototype for uh red fires iterium tracer and uh this mostly conclude uh how we uh leverage get live tracer to implement our firewalls indexer and provide uh deep data for people consuming the ethereum uh blockchain And uh that's it. [applause] >> Thank you.

>> Thank you Micho for just an excellent presentation there. Um for those that aren't familiar here, uh there are QR codes that you can ask uh questions on there for it. And I think we have our first question coming in here. Uh give me one second here to just get to the right screen here. Um I guess the first question we have on the top one here is is this open source? >> Uh yes this is fully open source. We have our fires tracer in a fork of get and uh uh also a bunch of other branches that supports the BSC polygon arbitrum optimism and uh the full stack is open source and can be checked in our GitHub at github/streamingfast uh all the stack is there.

>> Okay, next question. Do you have data on how large the flat files are? >> Of course, it depend quitely uh based on the network. On Ethereum mainet, I think we are at 4 terabyte of flat files right now. I think on BSC, we are at uh 10 or 12. And just an idea on Salena, I think we are at 80 terabytes right now of flat files. >> Okay. And then are you building as a Beasu plugin? >> Uh I did not check everything. We checked quickly. Bezu seems to have some tracing capabilities. Uh, will it be a Bezu plug-in? I would like to hear more about it. Uh, maybe, yes, no, I don't know.

>> Okay, there's been a reordering of the questions here. Uh, why did you go for GPRC? >> GRPC. >> GRRPC. >> No problem. Maybe it's a quick typo. Uh, why uh because it's widely available across all languages. So, it's easy for people to create a client in the language they prefer. And, uh, it's a well-known uh protocol used by a lot of people. So uh it was uh go it was a good reason for us and it it's well supported in go also and uh yeah that's the reason >> okay excellent here um can you use the same index to serve all of uh ETH calls >> so there's no uh indexing possible in the flat files directly the flat files have all the state changes but they are

not indexed uh at some point we created actually a virtual archive node from those flat files so we record all those storage changes the balance changes and we store them in a key value store uh uh over in fact big table to to not name it and uh at that point we are able to run ETH call on top of that it's kind of a state provider interface on top of get that is able to resolve the state on a uh distributed database >> okay and then running all the questions through here is there a specification for the protocol >> uh I would say the specification of it depend of which part. Uh if we talk about the fires indexer protocol you

just saw the spec uh no details what more than that but there was a few field but that was the spec those two messages and uh for the actual protobuff model it depends on the chain of course but those protobuff model are open source so they are the spec of our block model for the different chains that we support. All right. Well, thank you. Please, another round of applause from the show. [applause]