Erigon 3+? - Mark Holt - Erigon AG

Transcript

So, what I'm going to talk about today is um it's interesting the conversation we just had was about databases because I'm going to talk a little bit about um why Aragon's different and where do I need to click this at? So, it doesn't seem to uh um how do I move right and left? Ah, okay. Okay, cool. If I click the right button, it works. Um, so basically what I want to do is talk about a little bit about the features of of Aragon 3 and what makes it uh different and it really is around for Aragon 3. For Aragon in general, we've spent most of our

existence thinking about file format and how do you move data from a database to a um to to flat files and then distribute it across the network. I mean it's sort of interesting that actually now lots of other people are talking about it. At one at one stage we were the guys who sat in the corner and coded this stuff and uh everybody else was doing different things. Um but basically we have kind of three things. We have what we call ah got a typo but QLSMDB model. So we've got a database which is like a log structured database that has um this B tree at the top of it effectively. For us it's a continuous database. It's just

like two things built together. Um and in our uh log structured format we have effectively uh what we call page file determinism which means that effectively actually I might as well click to and then the third thing is we have binary distribution but let me let me uh talk a little bit from a picture. So effectively what we've done is we've divi we've defined our database. So the file structure other than the um the database at the top here is always a consistent deterministic structure on every node that Aragon runs on

effectively. So um and part of the reason that we've made this split is so that we can distribute the data between machines in a binary format without having to change it effectively. Now the the and this is really the change from um Aragon 2 to Aragon 3 is is adding these this this this data for state as well as as well as blocks. Now why why have we done that? We've done that because Aragon 2 had a tightly structured format that we put into MDBX and although it was efficient, it ended up with us having large MDBS DBX data files that were um I mean they were

smaller compared to maybe GU, but for the larger chains we were working on um you end up with an 8 12 terabyte uh data file which is is has got two problems. one is it's not it's not very operationally easy to deal with and the other thing is in practice it needs an MVME drive um in order to to make it perform and in cloud environments where most people run NVMe drives are premium commodities. So actually running an Aragon archive was quite an expensive um activity because basically you had to have all these local MVME drives. Um, so we kind of split things into files and

when we split things into files, we basically made sure the file organization was static once the thing had been written. Um, which basically means that as I say the things don't change between between nodes. Um and effectively we've we've also got a model where we'll we'll keep things in the database and then we'll write to files and then over time we'll write to larger files effectively. Um so essentially as the data becomes aged it comes it it gets moved into larger and larger units.

Now the practical reason for that is that it gives you a smaller number of files and actually gives you a um a performance benefit between the file and the um between the file and the paging system basically. Um so um and as I've just said basically every node does the same thing. So you can take a file off one node and communicate it to another um and it will always it will always be the same which means that effectively we can share our snapshot data um with no further processing and you can run the whole chain off of it.

The only thing that you need to do is put the kind of what's now quite a small database on top of it. So, Aragon's distribution model is copy this data around using using Bit Torrent and then that gets you very close to the tip and then you just fill in the tip in in the database. Um, now at the moment we've got an operational distribution model that distributes these things every week. So, it's kind of it's better to sync Aragon on Monday than Friday because what happens is over the course of the week you end up with with having to do more database processing. Um and yeah, this is just a um this is just a picture of kind of what what does this mean big picture-wise? We kind of see it

as we've got um and also because Aragon now runs a CL and the EL we kind of have got this we've got this stage where for both of these things we keep snapshot files. So we're doing we we're doing this kind of process of doing the processing creating a shared binary format and we kind of see it as right actually what we've done is we compiled the chain because if you think about the way a um way a compiler linker works effectively it's taking the code and turning it into a sharable binary form. We've effectively done the same thing. Um now the advantage that gives you is is kind of two things. one we're making use at the moment is we can torrent things

around but what we also and we haven't got to working on this and I think I talked about this this time last year is we can also virtualize this because basically this is a binary format and you can load it. You actually don't necessarily need to keep the whole blockchain on a um on a single um node anymore. As long as your network is fast enough for you to be able to have indexed data and you go and get the bit you need, you've kind of got what I kind of described as an internet scale blockchain at this point because the the the data store no longer has to be, you know, what can you fit on approximate to a single processor because as long as you can as long as your network time is actually quick enough, um you you can

actually store it elsewhere. And what what is quite interesting since we've been running Aragon 3 is um our Bit Torrent clients now everybody's actually started operating the chain can move 200 uh gigabytes a sec no two yeah can can move 200 meg 200 to a gigabyte of data around the network per second which actually means that this is almost practical because um if you think about the um if you think about the the chain model from um you know there's a lot of things are doing the the the first the last 100 blocks I know there are far fewer queries across historical data if

you don't need much historical data you can probably go and get it across the network although we basically haven't got that piece of the system working yet I we've not tried it now the other thing I wanted to talk about so that's the whole theory Now, Aragon 2 to Aragon 3 is something that we've we've I think this time last year at uh at DevCon, I talked about it because we were just launching. Um but in in fact, effectively I think I've got the time scale here. Yeah. So Devcon last year was about November and we basically were we were on our alpha still which we launched in in in July and uh effectively we we you know in our minds by the end of the year we were we were

going to have everybody using Aragon 3. Um in fact it's taken us quite a lot longer than that to get from kind of this model that we had that we had working that seemed to work to a um an operational system. So effectively, yeah, we launched in July. By January 2025, I think we were happy to call it beta, which means it kind of mostly worked. And what what from then we started doing was um was was getting Aragon uh basically working, getting people to use it. Um the same time we were you we were trying to maintain Aragon 2 and we kind of hit a hard stop

for Peterra because we realized that we only had the Petra changes in Aragon 3 and we couldn't really backport them effectively enough to Aragon 2. So unfortunately we had to tell everybody if you if you're on Peterra you've got to um you've got to go to Aragon 3. So that that gave us like a bit of a bump in the road where we had to force everybody to use our new product which I think wasn't wasn't ideal especially when you can see um by August we'd got to Aragon 3.17. So um you can see we'd gone through quite a lot of fixing during that stage just to just to get to a point where I mean the good news is we got everybody onto Aragon 3. The bad news is we forced everybody to get on

Aragon 3 which was was probably not completely planned. Um but I think where we are with it now is Aragon 3.1 has got a couple of important features added to it. Um primarily was I described the um snapshot uh publishing process that we had. Um what we basically had in Aragon 3 was no real file versioning. So we had these snapshots but they were all they there was no indication on in in in the files whether they changed format or they changed content and you know in the bug building process what we realized is if you've got this binary format that you're delivering actually you've got a

version of the binary format as well because otherwise you're kind of constantly pushing changes to people when you you fix bugs because when you fix bugs you also potentially change either the content or the format. of the files that you're delivering. Um, so Aragon 3.1 fixed that and is probably a lot more usable version as a piece of software than than actually Aragon 3. Um, and the other thing that we we had we we we ended up discovering we needed to deliver was um when we originally um delivered Aragon 3, one of the things that we were trying to do was deliver it without logs and receipts. And that's because basically in theory when you run

you're running an RPC node um because we basically Aragon 3 fetches things at a transactional level we kind of thought it was going to be possible to reconstitute the logs and receipts on the fly for for RPCs. But what what we what we learned was from a from a RPC scaled performance perspective that doesn't really work because you're you're forcing the node to re-execute everything and that's actually quite a load on the um it's quite a big load on the on the CPUs rather than um rather than just delivering the data. So um you know there's a trade-off here because basically receipts and logs almost double the size of the stored data. So,

um, effectively the way we've delivered it is these things are add-ons because if you're running a node and you don't particularly care about log provision or the bugs associated with it or the fact that it's unreliable, you can just run the model that we've got and it saves you a lot of data basically. So, it's it's a trade-off. Um, so Aragon 3.2 2 is a feature release for Fusarka and then we've just released Aragon 3.3 which is essentially implementing um archive get proofs which was a feature that we didn't have in 3.0. Um and again this is a you know this is a cost. It's kind of it will become a

a plug-in feature you can turn on because I think uh I think for ETH mainet our archive size for get proof is 4 and a half terabytes I think. So um you know it's quite a big delta on the 1.7 terabytes that we have for a for for an archive. So if you really don't need to support historical get proofs, you probably don't want to carry that 5 terabytes of of disk. So you know that's the story of us getting 3 point 3.0 out. I think we probably have spent a lot more time this year working on it than than we expected to. Um and why did it why did it take so long? Um

so one of the things with with Aragon it's it's used for Ethereum but it was also we we had a support contract with with Polygon and Polygon is a lot bigger than Ethereum. I mean it's quite I mean the the talk all earlier was talking 2x the Ethereum state. I think Polygon is probably three or 4x the Ethereum state. Um and it's, you know, because it's 2 seconds a block, it it goes a lot a lot faster. And I, you know, I was computing it's like it's 50, it's about 50 mega gas. So it's it's a block every two seconds. So it feels a bit like Polygon is at the moment where where Ethereum plans to be

in two years time. Um, which is interesting for us because it kind of says, well, we can probably support the growth because essentially it's just getting to where something we know worked was. But one of one of the things that we did learn was the whole release process of software is a lot harder once you get these big changes. So, it's not just how do you maintain it operationally, it's like the whole testing process is really quite slow. If you want to make a fundamental change to your um your tech stack and you want to make sure it works for the whole of the history, this is quite a big um endeavor and you end up waiting a lot of time to find out whether you've got a bug or

not. Um, and I think that the other thing for us is we kind of spent a lot of this year trying to improve our processes cuz um, Aragon has probably a rightfully a a um, a reputation for being it it saves you data. It can be fast for RPC, but it can also be um frustratingly unreliable because our change processes are are like we check the stuff into Main and and expect it to just work. Our QA processes to be frank up until this year have been have been ad hoc to say the least. And also kind of our our approach to software

engineering has been well, we're an R R&D shop. We'll try some experiments and we'll leave them in the codebase. um which makes it kind of actually difficult for us as a team because you get all of these things that somebody did three years ago that that's left there. So what we've been trying to do is just improve our QA and software engineering processes because now we've got kind of Aragon 3 as a product. We want to make it scalable and and stable. Um which is a bit of a culture shock for us as a team. Um but you know it's kind of we we feel we're in a we're now getting to a better place. Um and so question is was it you know was it worth the change because this change was

and the reason I'm mentioning this is I could hear conversations about moving from moving state out of the database into storage and I think although um although it's a you know it's a system step and the performance implications the you know our experience of this is actually that transition out of the database can can lead to a whole set of edge cases that you may not be you may not realize exist until you um until you do that and and you get you get things you expect to work and then somebody does a historical query and it doesn't return the result you got. And then when you when you interrogate it, you realize

because actually there's a bug in the code that's retiring and pruning data um essentially. And you quite often don't see those bugs until you know sometime down the line. And as with most most bugs, the problem is the further away it is from the actual thing happening, the harder it is to discover. So um bug chasing is quite a difficult thing before. I think the other thing is that we believe it's worth it because we think actually with Aragon 3 there's a whole new flexible way of working in that because because we're now not worried about keeping the data in the database and we know we can distribute it across the network. We've got the potential to do a whole set of more flexible things with our node than we did before. Um

somewhat frustrating that we haven't started doing it yet because we've been trying to make the node work but actually now we're there. I think there's there's a number of opportunities that we think we can exploit. Um and uh yeah the ne what I want to do now is say okay fine what's one of those opportunities um and it's a bit timely that effectively one of the one of the things Aragon's always been quite bad at is at the tip performance and because we got more flex um so Aragon 2 was like a big batch processing machine which is why it didn't really work very well at the tip because you needed to run the whole batch over and over again. It was very

quick at that. So, it was very it it it was very quick at basically or comparatively quick at at at getting to the tip from zero. You know, it it kind of would take a week or so or it would take for for um for polygon it would take like only eight weeks. Small amount of time. Now the thing is now we can distribute the stuff our chain binary. Um it takes a few hours to get to the tip. So you kind of get a slightly different attitude to life. You don't like start the thing and then go I'll come back tomorrow and maybe it'll be finished. You kind of get in you you you you so you know to start and to start an Aragon node on a Monday. It's

different on a Friday because it's it doesn't it's not quite at the tip. it can take you it can take you 15 20 minutes to start. So now it's like dealing with a normal software product um as as opposed to a node. Um but as as part of that we've started looking at kind of what how do we improve our our execution at the how we doing for time? >> I'm at it. Okay fine. So um just very you know so basically we've just looking at how we execute and we've we've put in place a parallel execution engine which we've just about got live which is great because now we can plug that into it.

That's not run out of time. >> Excellent. Yeah, I know there's just more content than we possibly can cover in a period of time and actually you've got one of the largest question sets I've seen yet in there. >> Oh dear. Well, actually I'll keep talking. [laughter] But unfortunately, I do want to keep us going in today's agenda. So, please a big round of applause for Mark here. [applause]