DappCon 2023: Making your node say your transaction history and portfolio balance - Willian Mitsuda

Transcript

so hello well I'm William and I'm here today to talk about your proposal on how to take your transactions history and Balan s from your noes and how you can do it first a quick introduction oops quick introduction about myself I created author K over two years ago uh who here in the a audience knows what the authors can isuse yeah good well for for those who don't know a scan is a self-hosted block explorer that you can room at your own home but today I'm not going to talk about that

specifically I'm going to talk about what I think is the next big thing that uh will bring real uh usage to ethereum clients uh we don't think that begging people on Twitter to ruin their own nodes will is enough reason for that or expecting expecting that they will just run nodes to help the network that's not enough motivation right uh I think the the answer is actually very simple we need nodes to provide data that users actually want something useful for

users and we think we we think that are failing miserably on that mission uh in order to illustrate that I'd like to tell a quick history about personal history back in 20 2018 I came in contact withum and the first first thing I tried was to R my own notes I heard of something called Miss wallet I guess most OG guys know what it is and I decide to install it and syn my own not uh some hours later I just gave up because it was taking so much time and I didn't understand why it was taking so

long uh maybe I was doing something wrong but uh so I just decide to use some hostage service which in 2018 meant using just inur okay great a few clicks later and I got the test node working and then came my second disappointment I did some test transactions some transfers and try to find a way to search for my own transactions and just to discover that the standard API uh besides using this thingal Jone RPC that no one else uses also don't have such a basic feature so back to today uh what's the scenario well

selling apis has become a business with we have many offers from different vendors every node provider has its own proprietary implementation and this is bad for consumers uh we have you have vendor locking for uh for each proprietary API implementation uh well the scenario today is uh it's the same basically the same as 5 years ago but now you can pay a little for closed closed sear Solutions we think this is wrong and what is our

proposal we have published a new API spec for what other would need in order to support getting users token balances and transfers and we have published what we call Au SC API V2 and this the design of the API was driven by author SC requirements but the end goal is not to be auor comp on it so that's why we are publishing it right now and try to gather feedback from the community it consists in a new set of custom stages that will generate all those data inside theone

database so it's a it's a reference implementation inside and for right now it has already very low overhead even though it's not uh the code is not optimized the the size the database format is not optimized but still it takes over one day to in a moderate uh Hardware to generate all this data the first time and also the the disk space is about 6 60 gigabytes or over 3% of Standards AR not for for main net so now let's take a quick look in

how uh how this API works and what does it provide first we have what we are calling the contract classification framework which is a custom set of custom stages that will take all deployed contracts and classify them and let's say okay they are there's a contract deployed it's an rc20 RC 721 or it's a minimal proxy uh that that's what it will do uh for now there's a small sets of uh classifiers but uh uh it's flexible to for you or and for us or for anyone to

create a new class classifiers and plug it into the the the the SN syn Loop uh this is a an example on how it works uh here is an example for getting the erc20 data so first we have a count methods which gives you uh let's say you have a main night nodes how how many uh erc20 tokens it has identified and you have a second method which is give you gives you a list of earc contracts so you have a a zerob based index and Page size and it give it giv

you gives you the the token addresses and this is how it looks inside Au K it it consumes this API and shows this fancy uh there is a new uh contract Explorer uh page where you can uh see in real time all the contracts being deployed and in this case uh all the erc20 tokens that it it identifies but of course it is it's is an API so not only others can can use it but your uh your own software can make use of it like you for example uh use some proprietary your C20 API from some node

provider and we have variations of these methods counting list for uh right now for rc20 for nfts for uh ERC 46 26 volts and RC 1167 minimal proxies the second feature we have now is uh the token transfer uh token transfer search API uh which gives you given an address uh all the transfer uh all the transfer transactions associated with that

address so if you receive or you send any token it gives you the the transaction list so again we have the transfer count method where you pass an address and then you have the transfer list where you pass the address uh zerob based index and Page size and it give you gives you the list of transactions it's important to say that that sear results are per transaction not per block so it it's sces no matter if the chain you are scanning has Bigg blocks or smaller blocks you get passionated search results per

transaction and this is how it looks like inside auor SC so it consumes this API and shows you this fancy pated search results list and the third feature we have for now is getting token Holdings per address so giving any address for example your personal address it gives you uh which tokens uh you hold and by you hold I mean uh which tokens uh have some uh transfer event Associated from or to that that address no matter which token uh and the

problem we are trying to solve here is uh there is an infinite number of tokens deployed inside your your main net for example so in order to get the balances for uh your address uh worst case scenario you would have to iterate over all those Tok tokens and get their balances so this this is kind of a filter you get all transfers associate with your address and then in a second step you can Quire the balance themselves and this is how it looks like inside scum this is by the way for vitalic address so there is a lot of spam tokens that is are already hidden

but there is a hardcoded list of uh White list of token addresses and we use this API to get this list and then we query each one in a timely manner so now about the future of those apis well even though it is it already works very well it is it does not take uh a lot of time and a lot of dis dis dis space we we think that we there's still room to optimize it uh we want we may explore using the Aragon 2 torent snapshot format so users can like for example nowadays when you

inst Aron it downloads the the headers blocks and transactions uh via torrent so we may uh Implement something like that for those index set data data so you don't need to uh regenerate it from scratch every time so you can just download some pre uh pre-made uh static files and you just uh run it to the tip to complete the data we we want to expand the the available indexers so there's a small set building we want to make more uh depend on what is interesting for other control display on in their

interface but we also want to allow users to plug their own indexers so we kind of want to make some stable public API uh it should be possible to hack the the existing code to add your own uh indexers but it it may require doing using some of the internal code of Aron that uh may change in the future so we we kind of wanted to make some stable interface so users can uh make their own uh texter uh they can eventually try to push via PR to other can or they can keep for for their own uh we want to

make make it easier to to H Aron code in pracy and as a small gift for you who are here uh watching this uh this presentation there's a test node so if you want to try those apis uh just please don't you seat otherwise I would have to take it down and that's it for now thank [Applause] you thank you are there any questions for William couple questions so first one is

uh which chains it runs on and second how do you handle like what you briefly mentioned all all those spam coins uh and currency so even not being vitalic you can get like couple hundred of different spam so how do I and they often look look like legitimate on even with the names and everything else how do I look at it see the value and can distinguish between the the different ones and how can I be sure of what I'm seeing and and what the data because if some scam says that I have 200 e but I don't and then I interact

with that not so much well it should work with any chain that are sys which means right now is out test Nets main netosis polygon uh but uh for this version I have only tested in sepolia test net and on May net uh because of lack of time but uh it should work if it doesn't work you feel free to open issue and I'll try to investigate it and about that second question about the spun tokens right well it it depends on which classifier you are talking about for example for ERC

721 it requires rc65 so if you we we try to what we do is we stch the evm inside the the the Aron stage and we load the contract uh into the stage and we we call the AVM and calling the supports interface which is the rc65 uh standard for uh for the N nft standards so we if if it Returns the the proper the expected results we say okay this is a this is an nft contract for rc20 it's more complicated because there is no standard it does not

implements RC 165 so we try some kind of duct typing so uh it's kind of a risk so we HDM we load the contract and we call the methods name in decimal and uh symbol and if the result is something like look like a think okay this is an this an erc20 token but even so there's a lot lot of FY contracts that they return some junk on on the string results right so we we have some uh not so beautiful codes that tries to filter and uh remove those junks of it's junk

okay this is this looks like a near C but it's actually not uh we we were able to do a good clean up in the current code but there is there's some false positiv is that we are trying to uh try to make it go away automatically thank you um you mentioned that you could add um your own indexing um is each index its own index or does it all share a list of addresses and then have attributes that I can add to that index

because it seems to me a lot of the ERC 721s are also 20s so you wouldn't have two separate indexes is that true or no uh yes the first there there are two kinds of uh uh uh major data C first is the the contract classifier so uh given Address given that code which properties this code have so it can be a NC 20 can be a 721 it can be a Min proxy and some some attributes can be can occur simultane simultaneously and there is there is the second uh major data structure which is

relating this uh some address with some uh some of these major uh contract types so uh occurrences of this address in the context of an nft transfer so this is another uh separate data structure and it can you can have you have multiple occurrences of these data structure so you have a transfers for year20 transfers for year c21 you may have other kinds of data structures which we still don't have but uh I think it will be

useful uh but that that's that's how the code is right now uh like I said it's not optimized I'm still unsure if it those some of those data structure could be collapsed and more optimized way but for now I I still trying to see the whole picture uh driven by the needs of other SC so that's why I didn't didn't bother me to optimize it a lot but actually first make it to work uh how as a user I want it to be anyone else no okay so thank you William thank

you very much