Optimizing for Noise: The Final Disk Space Efficiency Frontier for Archive Nodes? - Willian Mitsuda

Transcript

Okay. Hello. I'm William. Thank you all for having me here. I work for Argon and my talk is about the future of archiving nodes and some ideas on how we can scale uh state history storage. Well, as you know, scaling is this eternal battle between uh reach try to reach users uh demands versus increasing limits uh chain limits to levels that everybody feel comfortable with.

But as you know scaling is oops but there's a lot of chain activity which is based on temporary trends that generates a lot of chain uh activity for only a short period of time. uh however this this data keeps on onchain forever and that's why I'm calling that's what I'm calling noise in the title of the presentation but uh actually I think we could call it junk so optimizing for junk uh the outcome is never increasing state size which by itself is a big issue but in in this presentation I'd like to to

I'd like to emphasize another aspect which is the state history size uh which affects everybody that service history as part of their business like indexers or not providers. Uh also it's important to say that the goal of this presentation is not to uh advocate against history growth but uh embrace it and try to think about the ways to uh keep it change size growth under control uh long term.

So how we can keep size under control? uh one possible approach is to start looking at real data, real life data and apply a specialized encoders to it. Let me give two examples based on things that I've I've researched uh recently. Uh first I'd like to talk about index compression uh in in Argo because it's a mainly archive node. We need to keep track of when in the state history uh some accounts or storage slot change values.

Uh this chart shows mainet data. It's a histogram of how many touches in history uh giving key has. So a key could be an account an account address or some storage storage slot or a code hash. But you don't need to understand the entire the entire chart. The important the important thing that I want you to uh understand here is this blue and red bars. Uh the last bar the blue red sections they represent the amount of keys that have one or two occurrences in the entire history. uh notice how there most of the uh occurrences in in inside the argon index

and this is how it looks uh when you take key into account space uh disk space. So note again notice how the uh the keys that only have one or two occurrences they used to take this this is a previous is a node version of right so they take most of the disk space disc index disk space usage uh what what it gives us an insight that probably there should be a opportunity for uh an optim optimization that targets uh very short sequences this and that's what we did in Argon recently. We we used to apply we used to

use specialized encoding for long sequences called alias funnel which is very optimized for let's say you have a a very popular address or some let's for for example a unis swap liquidity pool that it's changed many times per block. So you need to keep track of all these u transactions that that that they they change some something in the states. Uh but that alias funnel uh has some uh some big overhead for uh for very short sequences. So in this case it's better to just uh store it for just use a simple of numbers. Uh and that's what we

did. Oops. So we changed the our index model to use a mixed strategy. So for long sequences we we still use alias funnel but for uh very short sequences we just do a simple uh array of number serialization and that was the result. We we reach it. So we can see that the blue and red sections are now much smaller and now we have much smaller index size and also a faster access time because uh there is no uh overhead into uh internal data structures just to to read the data

points and these are some reference numbers uh of a recent of a recent version of Aragon. Please note how this very simple optimization based on uh looking at real data, real life data uh made us reduce the index size by over half half in our chains practically. Um this is the this is the uh index compression. I I'd like to give another example which is something that I also have studied recently which is uh storage the duplication.

Uh one thing I realized while studying chain data is that we have over 10 to 20% of duplicate storage history values and why does that happen? Well uh thanks to standards uh in real real life data we have a lot of magic numbers uh start on chain uh one probably the most famous this zax fff which which means infinite approval for tokens and another uh popular one is you have popular token addresses uh ref cross reference cross reference it in between contracts

which is less common but also I think it fits in in this category of duplicate storage data. Uh this case is interesting because uh uh 10% may sounds uh something big enough to be interested to working on and try to optimize but uh we actually didn't optimize it yet because in absolute values it's it's still not not much compared with the the entire size of an archive node which in in case of argon is 1.7 terabytes uh according to data from September, two months ago. But this is something that we it's

interesting to keep a look at because maybe depending on how the chain grows, maybe in a few years this 10% may represent something uh very relevant that we we could work on and try to optimize and get some uh very juicy uh reduction. And in case of argon it would be very easy because uh uh especially in case of argon we from uh starting from argon 3 we now store all the history in static files. So we could do it very easily by let's say reprocessing our uh historic historical files uh removing this data and changing the indexes to

point to this semi offset inside this static file. So what is the conclusion of this two quick quick examples I I gave you here? Um from what I showed you today, I think one possible way to uh keep optimizing archive nodes long term is to observe real life data and based on real examples of try to identify data patterns of used by very popular contracts. uh it should be possible to craft specialized encoding strategies and keep keep data small and also another thing we we should try to

explore more is exploit immutable data data formats because once you know that data is immutable uh it can unlock some interesting strategies like for example the data the duplication that I I mentioned For reference there there's a recent Argon there a recent article I published on on blog which uh the charts I used in the first example were taken from here. So there's a more uh extens extensive explanation of what what they did here.

So if you are interested please take a look at it. And that's it. Thank you very much.