How to build the next Etherscan by Sina Mahmoodi

Transcript

hey everyone my name is C this is my first time uh in E Dubai and also in Dubai and wow it's a it's a great city except maybe the those two days when uh all uh chaos uh happened uh so yeah I work at the ethereum foundation I'm working on the go ethereum project um and yes first thank you so much for having me and I want to talk I want to uh show you how you can build the next ether scan uh before I get started with the talk I want to highlight that U duai loves ether scan I love

ether scan it's a great product uh this is no no way criticize criticism or anything but and there's always a but um two months ago recently ethereum had its Network upgrade Cancun and uh most people were excited me as well about the low layer 2 fees uh which is obviously great but in this uh in this hard Fork there was another tiny change or not so tiny maybe that connected the execution layer to the beacon layer to the consensus layer and um in effect it it comes down to adding a

reference to the execution layer block um pointing to the beacon layer and it's called the parent Beacon route so I was doing some development and I wanted to know the parent Beacon route of a block and a open ether scan as usual and I'm looking for this Fuel and it's not there and in that moment I was thinking man I wish I wish ether scan was open source um obviously I mean maybe some of you don't know but ether scan is a close Source product um I think this is how du why would look like if there were if people were using many there were many options open source blog explorers and this is

how it it looks like when there's only one Clos Source product that everybody is using so you know God sent us a sign guys we we need to we need to up our game um and I want to send appreciations here to blog Scout and aor scan so there exist uh open- Source blog explorers out there and in fact I saw on the schedule that somebody from blog Scout is presenting today in the afternoon so I'm excited to to listen to that talk and I'm sure that he can attest to the fact that building a Blog Explorer is hard and here comes the gist of my talk which is a new future we added in G that I think will make it easier to build a

product like uh a block Explorer it's a and other data indexing Solutions and I'm going to read this out loud here so this feature allows you to stream observable blockchain data out of gu programmatically and in real time before I dive into it I want to talk about my good friend the Json RPC API maybe you don't know exactly what it is but uh maybe you heard this guy complain about it on Twitter and you were wondering what it is well I'm going to tell you because the Json RPC API lets you read data about the blockchain and this includes your uh account

balances your contract State transaction receipts and whatnot and it will it allows you also to write uh to the blockchain namely by submitting transactions and the Json RPC API is great despite the complaints I think in fact it's so it's so good that it's used in almost every layer um of user interaction so up from the point that you're using an app you're using a dab sorry this dab relies on a web 3 Library like ethers or VM web 3js talks via jsonrpc to the wallet wallet in turn

talks via Json RPC to to the provider or the El client and so like all the way from the end user to the El client you can see the same API is being used in my opinion this is this coupling is a bit too tight I think we should break up the API and have something special for each of these layers but anyway that's a whole other can of worms I don't want to open that here today just want to say that for most people um jsonc server is perfect uh but then again block explorers are not your average user they have different data requirements than the D flipping nfts every once in a while they need they

need efficient Solutions and um and I think the Json R API is just not going to cut it so you might ask okay what are the what are the problems with it I think the problems with Json RPC the two the first two problems are Jon and RPC um Jason well I what I mean by that is um you're basically tightly coupled to a serialization scheme and Json is great because it gives you wide compatibility all across the ecosystem but uh you don't really need that when you're developing a data indexing solution you can opt in for a more efficient

solution and and RPC I think in my opinion is not great at at combining ing different resources and batching batching requests together and this is from my experience from two years ago so I was doing some experiments uh on the performance of the jsonrpc server in gu and in particular I was curious how it performs against the graph coel API that we also support and maybe not many people know here that gu also actually has a graphical API I thought not many people know about it I thought it's undiscovered it has great potential and nobody is using it I did have a rote

Awakening though because at some point I made a change there um and one day when coincidentally we were together with a team we were having nice dinner outside we get an angry email from one of the exchanges that yes there was a bug in the graphical uh API and they almost lost a lot of money because of us um so we had to jump on a call at midnight so anyway at this point I realized people are actually using the the graph server some some people not many and so yeah I had this hunch that it performs a lot better and especially for let's say when you're requesting different types of data and or long ranges of blocks and I did some experiments for example asking

for receipts for all transactions all transactions in uh in a series of 25 blocks and you can see that it just blows Json RPC out of the water around this time somebody opens a GitHub issue I think he was a me researcher uh because he was complaining that you know requesting a lot of story Slots of a contract over Json OPC is slow um so I was like okay hold on maybe I can do something about that I I wrote this Graal query and sure enough it's almost 10 times faster and then comes Felix my colleague and he says well you know what Cena our Json RPC implementation also allows

you to batch requests why don't you try that and I tried it and seems to have almost on par performance with with the graph C query I had which was very disappointing to me it was I was upset because I wanted graph coil to be faster and I did some digging and found a like unoptimized path that my code my query was triggering in the graph kill go library that we were using so I kind of worked around that I wrote another query and it was almost twice as fast the point of these experiments that I'm showing you is that to get Json RPC to work for your purpose you might need

to do a lot of tweaking optimizations for every for every specific query that you do and this is I think a problem something else that I want to mention is um that it might not the data model um of the Jon RPC might not always fit your needs as a block Explorer it was built for the mostly for the normal user who wants to interact with the chain um and for here I want to bring the example of utter scan Now utor scan is one of the open-source blockchain explorers block explorers that I mentioned it's built on Aragon

and uh the creator of it realized that they had to add a bunch of new API methods to get the performance that they want and also the data that they want for example they had to add a method to get the details of a block now you might know there exists a method like that we have e get block by number or E get blocked by hash but they just need slightly different fields and so they had to add their own method for this purpose or uh to get the creator of a contract this information is not exposed via the Json RPC server so they they just had to add a new method for this

purpose and with this I want to end the rant about jsonrpc and go back to the gist of my talk which is again um this feature that allows you to stream observable blockchain data out of gu programmatically and in real time now this time let's dive into it so first of all we call it the live Tracer and what it what it does is it lets you write a plug-in in go now unfortunately this is a statically linked uh plugin because the go ecosystem is is not letting us have Dynamic dynamically loaded plugins at

the moment um so basically you would have to compile gu together with your plugin but the good thing about it and I want to highlight this is that you will not need to Fork gu and we will see later on why this is so important um the data model here is push based versus pull pool based so in jsonrpc you always have to ask okay give me block n give me block n plus one give me the receipt for transaction all the time um and you will have to first you would have to wait for your note to finish sying and then you can ask this data of it whereas uh with with the live Tracer you will get all of these events sent

out to you it like broadcast to you it also has a rich data model because because it is a go plug-in so you have basically access to the data structures that we use in the GU source code and um you can can pick and choose the fields how you want uh process them as you want and then get them out of the Noe now how does it how does it look like in practice this will take a second so you write you wrote your plugin uh you compile it together with G and then you start a full sync your plug-in will receive events uh

as soon as gu starts up it will receive events like okay blockchain just started oh I saw a Blog I saw blog number one and I'm starting to process it in this block one there's no transactions so it's empty so I move on to block two now block two has one transaction I'm going to go into it I'm going to start processing while processing transaction in Block two I'm going into one of the calls it's a evm call here and such and such up code were executed gas changed the were logs emitted and so on and you will see like one call finishes the next call starts next transaction next block

and so on and you you kind of get the idea so basically as G is chugging along it's doing its thing it will also in real time give you all of these events and this information and all the while you will also receive these other events that are like for State update so basically any anything in the say changes the balance of an account changes uh contract gets updated you will receive events that um that of the relevant State updates so with this Rich data model you can not only display or index let's say the list of all the blogs and the relationship to one another because you

will also experience reorgs or the details of a transaction but also um more minute things like which logs were emitted during this transaction or this particular call and in what order or you can also see U as these are all screenshots from ether scan by the way um you can also see how the state was modified during the lifetime of a transaction or if you're one of those Brave users that clicks on this list button here and presses on parity trace or GI debug trace and you know about that and you see a page like

this this this kind of data is the bread and butter of the life Tracer very easy to get this data now we like life tracer was not developed with uh with a block Explorer in mind uh we started development together with streaming fast because they had this product it's called fire hose it's effectively a fork of G and it's adding a lot of hooks to diff like all all parts of the source code to to kind of extract this data that they needed and then they kind of store it on a flat play and have indexers index the data and put it some to some database and and

fire hose powers substreams and substreams is if you've used the graph some of the data for the graph subgraphs comes from substreams so these guys were maintaining a fork of gu for a long time and this is this was a sensitive Fork because it touched a lot of parts of the code and now they just have a plugin it's a one file and they they add together and and they can it powers for hose the other use case that that we were thinking is tracking The Ether Supply this is again a chart from ether

scan coincidentally um I'm kind of walking through all of the features that ether scan has and like showing you one to one relationship um it's not so easy to get the the total supply of ether from from a el client this this data is not exposed via via the API and and there's there's nuances to calculating the the supply I'm going to read out the description here from from the chart and it says The Ether Supply growth chart shows a breakdown of daily block reward Uncle inclusion reward Uncle reward E2

staking and e559 burn fees to arrive at the total daily ether Supply so it means when you're computing The Ether Supply you need to piece together a lot of different changes issuances and burns um to to get at it and you need to do de processing of the blocks uh to compute this and I actually think that the chart from ether scan might not be accurate because there is more there's more Nuance to Computing the the supply and here I'm going to bring two examples and they involve the sneaky little upcode self-destruct so when when you do a self-destruct you

have to specify a recipient address it means when the contract is destroyed where should the funds go here what happens if you spec like specify the contract's address as the recipient of the remaining funds well the The Ether is destroyed it's burned this is one way that ether is burnt that is not kind of included here and it needs evm processing of that block to find out that how much ether was burnt here it's same uh same situation with the second example it again involves self-destruct it's a bit different here because here the there it involves two

addresses basically one contract self-destruct and then a second one sends funds to it prior to the trans transaction ending and due to how evm operates due to evm semantics that ether is also burned so we have this plugin implemented um I hope this can power websites like ultrasound money um yeah ultrasound money like ether scan is also showing you the total supply of ether and how it changes over time and it shows the different parts of basically how how much was issued how much was burned and I know that these guys they have a they

also have a fork of gu they have this PR open on the GU repo it's been open since September 2022 so it's been there for a while and I can tell you something it's not going to get merged um and I was I was telling one of the maintainers about the life Tracer feature and what he told me was if it means I never have to rebase again I'm sold like maintaining a get Fork if you're touching core sensitive parts is really is really tough it's really hard when you go through a hard Fork so many things change and your code is going to break all over the place and it's it's a pain and this is why I think like this plugin approach is so so good because

you can just add one file not touch any other other parts of the code and there you have so guys this is all I had uh for today I don't know if I have actually time oh it's it's good because we kind of blew through we blew through this so I could take a question or two I guess check do we have any questions in the audience raise your hand please anyone all right do you want to have any final thoughts oh sorry I didn't see you uh this is something that I should have asked before but uh you say you have to recompile uh your own uh like

you have to link the plug-in in right but could we not somehow have uh something that would allow people to just hotload some uh like write write to default plugin that lets you hot hot Lo load uh Json sorry Json or JavaScript uh code to to specify what you want to trace and then off uh you know turn it off when you no longer need it basically so yes so we we do have a JavaScript engine but the overhead there is is big enough that I think it warrants native code so and unfortunately there's not a great plug-in system for for go at the

moment so there is there is a standard Library plugin um system but it doesn't work great it like it needs you to have exactly the same dependenc as guest like use all the same version of the dependencies and honestly like it's it's not reliable enough to to to use this so I think using a static static linked for for purposes like a block Explorer is not it's not so hard you really like you will probably just write this code one time and then compile it together with your source and and there it is any followup uh how about ebpf uh that was an idea by

Felix uh it's like a virtual machine that they start using in in the Linux Colonel as well maybe maybe that would have better performance something to look into I guess something to work on can the talk after the yeah like doing the conference weekend project we do this yeah all right thank you everyone thanks for having me all right everybody thank you CA