Panel: Client Performance - Pari (ethpandaops), Julian & Tobi (Base), Ben Adams (Nethermind), CPerez

Transcript

Okay. Hello everyone. Um, thank you for joining our first panel of today. Um, there's two things that we want you to take away from the panel. The first one is you're going to get cheaper transactions if everyone on this panel does their job right. Uh, the second one is once we're done with the panel, you get coffee. Okay. So I'd like to uh start by introducing you to um Carlos. So Carlos is um working a lot with state growth and understanding the problem of state um in this world and um followed by uh Julian. So Julian's with the base team and he's helping a lot with scaling base and we

want to today try and understand how they're handling benchmarks and things like that. We have Toby also from the base team and uh both of them together are working with yeah scaling base and we have Ben from the Nethermind team. Uh you might know him from the term performance is good. That's what his claim to fame is. So I think in the last year [laughter] sorry I'm going to try and troll them at least a little bit to keep it lively but uh I I love all of them. Um so in the last year I think we've spent a lot of effort uh scaling up our ability to do gas limit increases. You might have noticed that in the last few years at least on L1 mainet we've kept the gas limit at about 15 30 million in the let's say in

the last 7 8 years um we went from 30 to 36 million and the initial jump was already expecting a a move to 45 million but we actually had to pull the p push the brakes a little bit and only do 36 million and that was a bit of a wakeup call I'd say um there was a lot we had to rethink or look at in terms of scaling and I feel like we're at a stage about half a year later where we can talk about a lot of things that we've worked on to make those increases um yeah safer for everyone and the benefit is if the L1 does scaling well uh L2s inherit all of the benefits which is very very useful.

Um so I'm going to let each of them also give a one line or twoline introduction about themselves. Uh, I don't think I've gone too deep into what they do, but I'll start with Carlos. >> Yeah, I'm Carlos. I work at the stateless team at the EF. Uh, pretty new to the protocol team, but yeah, happy to be here and helping. >> Hey, I'm Julian. I work on um scaling base and benchmarking and more recently uh scaling get proof. >> Hi, my name is Toby. I'm a engineer on base. I work on >> Hi, I'm Ben Adams. cuz I work at uh Nevermind and I make things fast.

>> Perfect. Maybe maybe you start with a question for uh Ben. So uh you might have seen uh a couple of talks earlier we had Camille and Marik talk about their gas benchmark tool and a lot of things going on at Nethermind. Uh do you want to tell us a bit about the gas benchmark tool and what you guys have cooking there? Yeah. So that sort of that originally started um as a grant from Wilcoin. Um and we realized that everybody talked about issues with scaling Ethereum and there were like vibes. There were feels and um ad hoc measurements of um what issues were with scaling. Um but

there weren't any consistent um looks at that. So we went through and started trying to find all the uh what were all the lowhanging fruit, what were all the things preventing us from benchmarking so we could do it in a repeatable understandable way and from that the first thing from that was like the discovering that modx was like a a real really serious problem which is why we increased but halted at 36 initially and then um then that's expanded beyond that because obviously it was um we saw great value in finding all these hotspots. So we were like all right well we'll use that for Nethermind and you know improve all these these points and then that

grew again and we were like well it's it's it's great if we're scaling but that doesn't really help Ethereum. Um well unless you use the Netherland client which I recommend you all do. Um but the um so we engaged with the EF and we went bigger um to do it for all clients and then you know uh I think at Berlin interrupt we've formed created a leaderboard on all the um the things and then suddenly everybody started improving really fast. Um, >> Ben actually has a nice story about him not bringing a laptop because he was very very sure that Nethermind would end the week of Berlin with Nethermind still

at the top. Do you want to tell us how that went? >> Yeah. So, I just um I brought an iPad um cuz you know I didn't think there was much much work to be done. It would be mostly discussion and so forth. And then I think the the ref team turned up with 10 engineers that were just like hard coding and um by the the end of Tuesday I was right. Okay, I saw and everybody else was like creeping up. Um I mean we were we work in cooperative competitions so we do share our secrets with each other. We're not like you know um but yeah everybody was catching up far too fast for my liking. So on the Wednesday

morning, I went out to the Apple store, bought a uh laptop that has that has a German keyboard, which is really weird. Um yeah, and and yeah, started coding to >> Yeah, we really like that approach. Um none of them took it badly, but it seems that if you put a leaderboard and just put someone's name on first, every other team is really incentivized to like crack their gas limits. Uh that was that was a good primitive we came up with. Uh but the higher we increase the gas limits, we're going to run into state growth problems, which is something that Carlos is uh looking into. Uh yeah. Do you want to tell us what your biggest fears are? What what's in the pipeline?

How what's what's uh what can we look forward to? So we we started uh basically because all of the discussions in ACD as far as I know and I'm pretty new were pretty by based uh with respect to state like there's no data uh or there was no data to back up any of the statements like we are super worried and other people was like chill we are fine uh but no one really had any data to back anything up and it was pretty confusing and hard to like start taking decisions in order to okay what do we do? Um so we basically uh created this bloat net thing and we started like taking benchmarks and addressing all of these all of these issues. We saw that we could collaborate a lot with

Nethermine and the other teams to like help on scaling because indeed compute is not just the only bottleneck that we have in order to arrive to 100 million gas uh limit or whatever it is that that we define. And besides that um there's also the the question which is we are setting uh some kind of uh difficulty bomb or I don't know whatever you want to call it. Uh the more we climb up the gas limit the bigger the state will become. Uh and we don't have any alternative to that. And so the main worry in that respect which is not scaling is basically how can we make it as easy as possible for people to actually keep holding state. uh do we need to have incentives like like what what does need to happen so that people

don't just start abandon uh running full nodes uh especially when GKVM or any other sort of statelessness arrives and like literally there's no incentive to have uh data in your computer so yeah >> nice thank you um and yeah the next question I had was for Toby and for Julian um so Bass is also aggressively tweeting about increasing gas limits and also achieving it which is great uh do you want to tell tell us a bit about what your stack looks like. What are you guys worried about? Um why scale the gas limit so quickly? What's what's what's the thought process?

>> Yeah, so I'll go over the history of why we started scaling. Um so yeah, essentially we wanted scale to essentially have a lot of people use base and uh it's going well for the most part. Um but yeah, so the historical data uh historical reason for uh like why base was scaling and what problems we faced. uh first problem we faced was the L1 uh data availability. So at the time before PCRA and now Faka uh the blob space was three target and uh six limit and that we saw that we couldn't fail a lot of L2 blocks into the L1 and that caused us to uh try to look for other opportunities where we can

essentially optimize base uh blocks and move that into L1. Um and first thing we saw was okay we can compress the L2 data a little bit more into uh L1 and we uh worked on the batching uh algorithm. So essentially uh we implemented the broadly uh compression algorithm along with the op labs and that went really well. were able to still use uh some of the blob space that was there at the time before uh Petra and uh helped us uh scale base to uh I believe uh 25 plus megas and uh yeah after Petra we're able to scale up a little bit more. Um we also noticed then at that point that uh the state growth was uh pretty large at

the time of the GU archive uh the GU archive state size was about 55 terabytes and it was growing at 500 to 600 gigabytes per week and we were worried that sequencer wouldn't be able to run on uh a lot of the machines on AWS um at the time and so we decided to come up with a hybrid approach to uh setting up the sequencer. So we started with like a full node we had full node snapshots uh daily we would set that up and then from that point in time till tip we would essentially have the uh node set up as an archive node and that was able to solve some of the uh state bloat um problems for some time and eventually now we moved to uh ref. So we're using uh ref for the sequencer and

uh flash blocks as the arc builder. Yeah. Uh just in essence, we're using a lot of um different techniques to try and grow uh base. >> Cool. Do you want to tell us a bit about um how you decide if a certain gas limit is safe? >> Yeah, totally. So um our approach uh initially was essentially a reactive one. So um you know L2s are one of the benefits is that we can experiment a little bit more safely. um in L1 it's hard to roll back a gas limit increase whereas L2 it's literally like we change a parameter and we can go back. Um so that was the initial approach but

obviously that doesn't work in the long term like we want to make sure we're we're scaling to a safe gas target before we uh get there. And so um that was kind of the inspiration behind um our own benchmarking tool. um we uh we kind of didn't know about uh Nethermind and the EF's initiative to do this or it wasn't uh to the point where we that we needed but ours is really focused on benchmarking storage operations for a very large state size. So I think we're like more than 2x Ethereum state size.

So um storage operations um it's a little bit more scary. Uh so yeah, that's kind of uh what we've been focusing on since the reactive approach. >> Nice. It sounds like there's a common theme and a lot of it is to do with how we dealing with storage, how are we dealing with larger nodes, um besides I guess making state access or state writing cost thousand times as much, what's in our tool belt? What can we do? Um so it it's always trade-offs at the beginning I would say like for instance uh with block building with between quot surrender to just having builders and

they build for us and we take advantage for that from that and we build around making it more secure whatever it is and we take profits in other parts of the protocol out of this. uh I don't feel really personally at least really comfortable saying well we let's just centralize state holding and keep it to RPCs and I know one or two freaks uh that want to have crazy machines at home. Um so besides all of these trade-offs which are really really hard to make uh what we are trying to do lately at least with the stateless team is try to see if we can approach state expiry on a more subtle and and step-by-step way. the state expiry for the ones that uh don't know which I

assume is almost nobody but it's it's just getting rid of parts of the state. uh so basically reducing the the database size and in that regard there's there hasn't been any proposal that actually has been widely accepted and welcomed by everybody and and I can understand that there they all have like uh problems issues whether it's in the UX or any any part and so what we are trying now to figure out is can can we start translating part of the storage responsibility into wallets of users making them hold just like tiny bits of data let's say like just leaves and maybe some proofs. Um can we also do out of protocol uh state expiry where we

don't need like hard forks and we can approach the thing a bit more slowly like similar to erafiles things in the state compression uh article that I think published this week or the previous one if I'm not mistaken. Uh so yeah, these are things that give a lot less improvement, but I think we'll eventually put a step on the door and we'll slowly make it easier for us to start getting numbers, see how users react to that, and then we can slowly make it better and and a bit more aggressive, I would say. So yeah, >> nice. And hopefully that's also something we can test out on L2s that might face a problem before L1's. Maybe there's a there's a collab working there that could work. Um, let's assume that state is solved for a bit. What else are

we expecting to be problems? What what other things are on the pipeline that we need to care about? >> Um, I' I'd flip it around and say, um, the the inverse is, okay, there's there's a lot of lowhanging fruit that are problems to scaling, but there's also a huge amount of things that are overpriced. Um, so we can bring those down. So that's sort of one of the glamster focuses um is reorganizing everything so that the the pricing matches. So you know if you're doing um including the um base transaction fee.

So 21,000 gas that sort of comes down to 6,000 and then you start building on top of that. Um yeah. So >> do you want to go a bit more into why we're repricing um besides the obvious? So we thousandx all the cost. We thousandx the gas limit and we say we scaled Ethereum by,000x what what's the idea behind repricing? >> Um it's it's to allow people to do more with the same block space that we have.

Um I mean obviously there is the state problem but that's that's a secondary dimension. So all the, you know, mathematical operators, um, as I say, the base transaction fee, um, they're all overpriced for what what you can do. And in a way, dropping the the price of those, um, op codes does ineffectively raise the raise the state price um, relatively, >> but that's probably another big factor, right? So we still re rely significantly on the RPC to communicate with the node.

That's pretty much the only way a user would. Um how would you expect a user on base to be communicating with RPCs? Do you just expect the RPC operator to have bigger nodes or what's what's the what's the what's the plan? >> Yeah. So it depends on what layer they're trying to access. So if it's flash blocks uh we currently have the uh roller boost proxy websocket proxy that we can uh users can essentially use to access flash blocks at 200 millisecond block times. Um but for the uh the main canonical blocks uh we want to be able to have people just use you know RPC providers if it's uh Alchemy or Quicknot or etc. Um I think that just having

multiple RPC providers exist and they have uh more optimal uh systems would probably be the uh the move there. >> Do you guys see some way where we can move more and more of history on slower moving discs or if we can have different tiers of storage? Is this a direction you've you guys have thought of? >> Yeah, definitely. I mean uh there have been many proposed like tiers of storage like you can run a full node now you can run an archive node and you can get rid of history um like re as of recently. Um so I think one of the big things for us is we need to be able to prove the execution of a block um which relies on uh archive like proof state uh which is

like all of the changed tree nodes. Um, and so one of one of the uh areas that I've been working on recently is um integrating that storage into ref um and basically just having like a separate storage that's slower um that can be used to serve uh proof requests and essentially uh re-execute a block um which allows us to prove it on L1. Is this possible with one of these extension frameworks that Reth G have spoken about or um h how how is it that you're able to do this?

>> Yeah, exactly. So, we're working with uh Optimism on a Ref extension that will uh override certain proof um RPCs and serve them from a slightly slower storage. when Netherland extension. Uh well I mean we do have a plug-in system but we are looking at maybe um adding the refex format and then we can you know people can easily move over and join uh join the never mind.

>> That's good. Uh I mean we do need client diversity with this as well otherwise we struggle with having just one client that's hyper optimized for RPC. Every RPC provider relies on them. Suddenly you have a bug and it feels to every user that there's a client bug right? So, it's nice that the architecture is generalizable and then you might have an implementation in one client that you you're able to port over with multiple clients. Um, okay. Then we have a couple of questions out there as well. Um, so what's maybe the first one's great for Ben. Um, what's uh Nethermind's recipe to hit peak performance? We all know that Rust is the best language that ever existed.

Um, how is Netherm able to to beat the benches? Are you benchmarking? [laughter] >> Um, well, I'd say I'd say .NET and C are not what they what people think of them. Uh, since they open sourced it and like over 50% of the contributions in the last I don't know if it's five or 10 years are no long external to Microsoft. um they expose all the CPU uh SIMD primitives um all that sort of stuff. So it's it's a very different language um than you probably expect and you know you can you can just not

allocate memory and then you don't have to worry about a garbage collector. You know there's all sorts of all sorts of different things. How does the workflow work then? Do you look at the benchmark, notice what's the slowest, try to trace the client, and then figure out what what goes or do you just do you stare at the client code and then the optimizations come to you? [laughter] >> Um, so I I have a client running at home 24/7. Um, I'll normally um trace uh trace blocks around sort of in the Asian hours because they're much more interesting cuz the gas generally drops and then people start doing very

interesting transactions. Um and um so I go I go to bed quite late cuz I'm in the in the UK. Um but then in the morning I will you know check whether came sent me you know an angry message why is this regressed why is this regressed and see if there's anything I need to uh to look into and that's part of our workflow for release cycles we make sure we have no regressions and in performance and all that um and then you know I'll look at the traces see if there's anything that could be found see if there's any new issues and then have a look at our benchmark panels and see if anything we need to to look into or if some clients, you know, maybe pushing

us out of the top slot um and something needs to be done um in a in a good way obviously. >> Cool. Um how so on the L1 we're looking at scaling from let's say 30 million to all the way up to 100 120 million over the next let's say year, right? Um does base have a plan? Uh how how are you guys looking at it? Is it is it uh demand driven? So you notice that gas is being used so then you decide to scale or do you have a proactive scaling plan? How does this work?

>> Yeah. So like Julian mentioned earlier we have the benchmark in tool. And so whenever we before we scale we essentially try to check okay if we're scaling for 100 mega gasund 300 megastas now uh to 500 or whatever uh we essentially make sure that it's uh good for all uh types of up codes uh and storage types uh storage reads and rights. Um I think going forward with uh L1 going up from 36 to uh 100% said um yeah it's just better for base uh because we just essentially uh will allow people to have cheaper transactions and uh with uh Saka it's going to be even better too as well for uh being able to bundle a lot more L2 uh

blocks into L1. So pretty excited for that. Um I'd just like to to highlight a possible disconnect in terms of um use like similar client well use the same clients between L1 and L2 and um it's because on so on on an L2 they can use like peak hardware um you know it's like the sequencer and so this is why you'll see like nethermind will do like a giga gas a second on really good hardware but on L1 So you could on base they can use really good hardware but on L1 we have like uh thousands of nodes and we have like a base hardware requirement that's

that's far far lower than what you can push an L2 to. So that's why there's there's a disconnect in the benchmarks between uh L1 and L2 if that makes sense. There's however an interesting thing that could happen in the next fork, right? We have block level access lists and the performance improvement that block level access list gives you is as good as the hardware you're optimizing for. Does this mean that the L1's probably going to assume where we have a parallelization of like four whereas base can parallelize 16 transactions?

Is this linear? Do we see some sort of um falloff fallout? What do you what sort of optimizations are you thinking of? >> Um I think minimum >> wait before that maybe a brief intro to block level access list before you go there. >> Okay. So block level access lists um as part of the block it includes all the storage locations you know the reads uh well the rights more more specifically of all the transactions. So it's like a a change set that comes with the block which means you can now execute all the transactions in parallel out of order because you can just look back at any

state that's been changed by previous transactions. So we we can do full parallelization both of reading the data from disk and also executing transactions. Um yeah so it it should be quite a big unlock in performance but I mean this this affects the validators specifically whereas the block builder um who has to generate the block level access list they don't get those gains because you know they have to create it in the first place >> there was uh I think Gary uh from the GU team worked on uh have some preliminary benchmarks on block access lists with

some of the worst cases which were reads uh in big contracts to keys that were non-existing or basically had zero value and he proved that even with that we can already get if I recall correctly it was like 4x but I don't want to misquote it uh check the the thing but it looks like indeed will help uh on some of the worst cases we have which which is really nice >> but does this scale so if I have an 8 core machine will this show a 4x if I have a 16 core machine will this be 8x >> if I I mean you can ask him [laughter] but I don't want to miscode anything but it it's also related on of course on how good uh IO uh you had with your disc uh

but yeah I mean feel just make sure to check out his article I'm not sure if it's public or not but uh I'm I think it will be at some point so it's everything is there >> nice um so that also probably implies that state's going to get grow even faster. Um, how what's what's the plan for sync? I know that you've been working on some uh effort called bloatnet. Want to maybe explain bloatnet and how syncs looked there so far? So in bloatnet we just have uh we took Ethereum mainet uh well you took Ethereum mainet indeed and you shadow forked it and we just started to uh spam it as much as we could uh with really high gas limits until we got to around 2x mainet size maybe a bit more uh but

we kept the ratios in the state. So if we multiply by two uh account uh the account storage we also made it for the contract storage and we also made it for the by code. Uh we are also trying to make sure that for the worst cases we have so for example like really big contracts uh we also make even worse cases for those. So I think the worst thing we have in mainet now is Zen which sits at 9 something gigs uh which is pretty big. Uh so we are trying to go 2x or 3x uh into those cases and see how it actually scales. Um so this this has mostly been the the strategy on how to detect bottlenecks and then how we build something that we all can can use and I

think Gary's uh study with block access list also showcases something nice from bloNet which is we it's not just uh something that we can use to find worst cases or or this kind of stuff. We can also check if the EIPs we are proposing are future proof and will actually help us when Ethereum is in a much uh different shape uh being bigger or I don't know whatever it is. So I like also this this part of of making future proof uh EIPs that that we can test >> And um how does this look like for base? How do you guys do sync? How do you sync the L2 chain? Have you noticed that there's been issues? Do you rely on snapshots? How does this work? Yeah, good question. Um, I think we probably

rely more heavily on snapshots than L1. Um, but, uh, our main goal is to make sure that, um, other people can run validators on, uh, hardware that's available in cloud providers. Um, so I think just being able to catch up to tip uh, like I think we have weekly snapshots and then catching up to tip may take a week or two weeks or something. Um, so it's it's uh a little bit it's definitely slower uh than L1 to sync up a node. Um, but that's kind of the price we pay for higher performance.

>> That makes sense. Um, and there seems to be another question about um, how are we making sure that all the clients move with us as a group, right? Uh, at what point do we decide that a client has to be left behind? Uh, the question specifies EthereumJS and Trinity. Um yeah, maybe you want to take that Ben. >> Um I mean we do share amongst as I say there is a competition but it is a cooperative competition. So any breakthroughs we're at the end of the day we're all Ethereum. So we do share you know any new breakthroughs we've made um we share that between the teams.

um any now that we have this um benchmarking system that's going on continuously um that we talked about earlier um you know we we uh point out you know ask clients if they need any help to improve things and you know what what can we do there so there isn't intentionally any you know we're leaving a client behind but I mean I suppose the community adoption so forth is not Yeah.

>> for us to decide. >> Yeah. I think to add on to that, we really love the benchmarking work and like I think that's going to be the key to making sure no client gets left behind. Um like just I I really like the the leaderboard concept and kind of like showing you know what what the worst case is for each client. Um so I think it's going to be also very similar for block level access lists. You know different clients are going to have different implementations of that and uh measuring that and publishing the results. Um I think we as a community need to do more of that.

>> That would be awesome. Um maybe as a follow on how many of you care about EL separation? Is it uh easier to optimize? Is it worse to optimize? How does this work? >> I can if you want I can start. I I I like the separation u but I'm also a docker maxi and the separation doesn't really bother me. But yeah >> I mean from a client diversity standpoint in terms of like bugs. So you have is it five CL clients, five EL clients and together that makes a combination of 25. So uh so you've essentially got 25 clients in terms of diversity in terms of like

having um hitting a bug that's going to hit all 25 combinations exactly the same is is is unlikely. So in terms of the resilience of the network it's it's very good. Um in terms of testing I mean I don't know that's not [laughter] that's not my area. Maybe you have a comment on that. I mean, yeah, a fewer clients would be nicer to test. However, there's this nice primitive of separation of concerns. Um, so the consensus layer can focus on consensus. The execution layer focuses on performance, focuses on state. Um, and that allows independent engineering teams to do what is right for their area of concern. Um, similar with um the code bases are separate. You

have different types of eyes looking at different things. Um, I don't know. I kind of like it. Uh, engine API itself could be a lot better. Uh, I think one of the pain points is often that it it just has so many edge cases that are hard to define that the nodes just look weird in weird states. Uh, also I'm not really sure why we did it as a JSON API. Probably we can use SS or something a bit more performant. But I think there there's probably things that can make it better. But usually the complaint comes from like a node runner perspective at which point I tend to question why aren't you using something like ETH docker or like a deployment tool like DAP node. Um but yeah there's a lot of questions there.

>> I mean I'd also add that the fact that they are separate components does enable like L2s because you know you're not using um the CL. >> Yeah. >> How does this actually work? who what what does the consensus engine of the L2 look like? I've I've never dug that deep into it. >> Yeah. So, uh for optimism, we can talk about optack. Uh it's essentially op node and op node uh sources um like uh blocks from or I guess uh receipts from different peer-to-peer uh op nodes. And by taking that uh receipt, we can then uh unravel it and uh look at see it in the L1 uh the L2 uh execution client. Um

OP node is stateless for the most part. Uh there's some safetyb stuff in there. It's slightly different from the consensus client like Lighthouse and uh Prism. Uh yeah, but it's for the most part like uh slightly different. >> Nice. Um there's another question about the side effects of uh raising the gas limit. I mean we've spoken a bit about RPC, about state, uh about benchmarking, but why were the OG core devs against it? Um what what why did we take so long to I guess take up these problems?

>> Uh I think there were a lot of unknowns. Um and also like hardware has moved on quite a bit. So, um, you know, you can you can now buy the same hardware at a mer. I think you needed a 2 GB SSD at the merge and that was like500 or $700 and now they're the same but faster is only like $100. So, you know, and CPUs have improved, memory's improved. Um and also now as I say we have the the benchmarking we're looking we're looking into very deliberately what all these scenarios are whereas before

um we didn't have that detailed data >> I think we also came up with a couple of new database types in the last years right um what's the and is there a tlddr for why uh Aragon and database looks so different from nethermines and that looks different from gets I'm not or who wants to take the database question. >> I'm not sure exactly why like what the reasoning is but uh I know like we the ref database makes certain optimizations like not storing the bottom branch uh of the state tree because those just change every time you change state. So reducing right amplification of if you write a single of a single key how much data actually gets written to disk. Um I

don't know if you uh yeah anybody else >> I mean MDBX is just stating trade-offs on on LSM based databases and uh I'm not sure you should ask G people for >> Could you expand on what those two words mean? >> Uh I mostly use the I mostly use the shortcut so I'm not I'm not sure I'll be able to tell. Uh my I I think that you should ask someone from GE for example or Nethermine on whether they would be open to migrate if it wasn't such a such a massive change. Uh most likely I know uh GE already migrated from uh rocks if I remember correctly to pebble uh >> level exactly. Uh so that if I understand correctly was a quite painful

uh change and probably it will be even more painful if they switch to a significantly different format which probably tackles a lot more places. Uh so maybe that would be the case but if anyone from G uh or someone that drives >> um I mean they they'll do optimize for different things. So like the Aragon format is uh for an archive node is incredibly compact. Um then for um what we store the way we store our data currently is very good for very fast for calculating state routes um but I know like but slow for accessing um for doing like S loads and and they kind of like

even out. So um so ref for instance has very fast uh s load access but then it has terrible state root calculations and they sort of like kind of even out in >> Okay. So basically sorry go ahead. Yeah >> that's interesting. >> Yeah. Um this is uh it's probably it's a work in progress. I don't know uh if we have anything to share specifically um but it's basically an optimized version of storing the state try so that it um is able to calculate uh state route quickly. Um it's it's definitely a work in progress. I think there's going to be

news at some point. I can't I can't commit to anything but uh it's definitely an ongoing effort inside of base to uh experiment with different uh database formats. It feels like we have a lot of names for them. So I've heard bonsai, forest, paprika, try hash and path. Are these all genuinely different things or are we talking about three different types of database? What's what's with the naming here? >> Um yeah, they're all different ways of storing data. um and they they achieve different different things.

There's no nobody's quite come up with um sort of the ultimate uh format that I'm aware of. >> Do you think there's ever going to be an ultimate format? >> Maybe. But I mean, if we change if we if we do something like vertical trees, then you know that that's all going to if you get too specific, that's all like that would ruin the the format. So, I don't know. >> Okay. Maybe uh there's there's one more question. Any recommendation for an open source benchmark tool? Who wants to shill their thing?

>> I think uh I think we need to unify on a single benchmarking tool. Um I we built ours because there didn't exist a good one for benchmarking storage operations on existing state. But I think that's quickly changing. So I don't think it makes sense to have five different benchmarking tools. um they need to work across all clients and support like a very wide variety of use cases. So um yeah, I'm excited to to work together on you know unifying all of the benchmarking tools to have the one the world.

>> It's only uh it's only recently that we even unified the test formats for the benchmarking tools which is great. Uh so now we use the execution spec test format uh which is also what we use for testing. So, it's it's been a process, but it feels like we're making it cleaner each month. Um, okay. I think that's mostly it. Any closing statements from you guys? Um, which client should we run? What what's uh what what's your I if you could have anyone in this room help with any one thing, what would that one thing be? >> I would say contributing uh to the open source like just community. Um it's been very helpful for base node base node to see like fixes for uh certain uh bugs and whatnot and it's I think just

awesome to have that ecosystem. So >> um I'd second that and say um open source is is very powerful. It's very leveling. Um and Ethereum is open to all and you can just do things. I would say uh it's it's probably easier to uh scale up if we go if we give up uh decentralization and that should definitely not happen at least in my opinion and the real challenge for these upcoming years in which we will definitely need as much help as possible is how do we actually do that? uh how do we scale and make the state much bigger but we don't compromise uh the access to it and the usage of it uh controlled by

a minority. >> Yeah, I think that's a great space to end it. Uh the future looks like higher gas limit. There's probably a lot of ZK stuff and we're moving in the stateless direction. Um but that also comes with a lot a lot a lot of problems. So, please start getting involved and um help preserve decentralization while we scale. Thank you for coming. [applause]