All Core Devs - Testing #74 (BAL Breakout)

Transcript

What happens then is we have well the main question is Do we want to consume bandwidth for this optimization? And this is of course dependent on how good this optimization is. Like what is the actual impact? of the batch IO like the pre-fetching of the storage slots. Awesome. Thanks for the update. Does anyone want to add something to what Yohan has said?

I'm also wondering because we had this discussion of how many cores we should use for benchmarking. Would this be something we should also discuss? What do you think Yohan? I think we should maybe just do that to get like a temperature check on yeah. Okay. What I think makes sense. Okay then just to Yeah, I'm here. Maria posted a message in the EthPandaOps chat asking if we have any data on this especially finding a way to remove like attestation effort and like everything related to the CL from the full node.

So Sam has been working on this. He has this eBPF based approach so that we're able to get like sub slot metrics. So you'll know exactly within us okay maybe a higher level. The problem with using Prometheus for getting this data is that Prometheus has a sampling rate and that sampling rate is not very great. You can reduce the sampling rate as much as you want but it doesn't give you the super fine grain information that we're looking for. So Sam's been looking into eBPF and we have this custom metrics collection agent on eBPF now and deployed on some of the Zalthus nodes. So he now should have all the data required for finding on a sub slot basis how many resources different

processes took up. He's of course not awake right now to share a lot of this information but should be able to reach out on the next day and give this information. Awesome. Yeah, this is This is great to hear and just for our context for everyone else so for the re-pricing of course we can we also need the ball optimizations there. But for the re-pricing we would of course be a little bit more conservative when it comes to how many cores we actually want to use for the benchmarking when it comes to re-pricing.

But then separately from the re-pricing topic we also need to find out how much we get from the state locations in the ball. So basically how much do we get from the batch IO. And for that we shouldn't be too conservative but target to be very realistic such that we actually have a quite clear picture of how much do we get and can then take a decision of removing or keeping the state locations. I see there are some hands up. Please go ahead Daniel.

Yeah, I I just have a question for for party about what I just shared. Can't we run the EL and the CL on two separate nodes to have cleaner benchmarks? You can. The problem there is it won't represent how most people run it on a real network and we have reference hardware and that reference hardware is meant to take up both EL and CL so Okay. Okay. Okay. Okay. Makes sense. I see Rahul you have your hand up. Yeah. So for ball benchmarks one of the metrics that we collect today is wall clock timing for for completion of the engine API

payload. I'm also looking into fetching internal metrics from clients. This could be time spent on reading the state from within the EVM. So that should give us more granular Rahul if you're interested there's also the tracing approach. So there is like a definition of standard tracing. I don't think anyone's really implemented it this way but it is something we wanted to push clients for implementing. It could also be an interesting way to capture the data.

Yeah, I think I'm building on top of the open instrumentation the thing that Stefan pushed. So I'm just trying to like standardize this for one client and then I'll publish my results so other clients can implement. I've shared the client traces repo. If you want changes there just make PRs and then ping me or patches and we can get them merged in. Awesome. Is there anything else someone would like to add especially has any has any anyone already played around with their own batch IO implementation

and discuss observed something? I know Stefan has been looking at it on the devnet but of course on the devnet we don't really have an up state to actually get those numbers and also we need worst case traffic instead of average traffic. So that's why it doesn't really work on the devnet. But yeah, so this would be great if we get those numbers as fast as possible such that we can take a final call. Yeah. The storage reads are in address order.

Um I'm wondering >> What? Sorry. I didn't They're in they're in address address order. Uh not access order. So they're not >> Yes, exactly. Yeah, so I'm wondering if that also would have an effect if they were in because essentially they're reading batch but they're reading in a random order compared to the how the transactions are going to consume them.

So that might also have an effect in that you might be loading you might be running the first transactions together but loading the storage for the the last transaction at the end of the day. Something to consider. Yeah, this is definitely something we have considered at some point. There were some arguments against against it. For example, it would make make proving things against the block live access list more difficult because you couldn't do like proof that a certain address was not touched for example by just proving the address that comes like say lexicographically

before and afterwards. I remember there were were some discussions around that and I know that we have benchmark tests that actually this distinct limits basically having the last transactions So basically having what the last transaction touches being first and then reverse that. So especially so that you if you implement batch IO and you just implement it in the order of where as it comes in the ball then you would load everything that you only need in the last transaction first and then the second last transaction second and so on. So we have

benchmark tests for that. >> We should we should observe we should observe it through the tests rather. Okay, that's cool. Yeah, we we should see the difference between like an average case and this worst case in the test. Of course the the hope would be that this would only affect it like minimally because batch loading should be very fast. But this is a good point. Yeah. Yeah, Emre Sen you also have your head up. Heads up. Yeah, just related to what Yohan said on the benchmarking side.

So currently do we have like the benchmarks to test batch IO um without using the devnet or not? I am wondering because this is something we would like to test on base aside. We would like to know if uh if we have real improvements from batch and IO because it is not really clear. So I would like to know if the benchmarks are ready or not. Yeah, I got my time on this. I think there's like maybe like a a confusion is not the right words but I think we should really write a read me on the benchmark status because like

benchmarking is something completely new. We started this benchmarking effort like one year ago. The progress is like very good but seems that a lot of people don't really know what is actually being tested. Like is it like a network or is it like the client itself. So this is specifically clients on a snapshot of like of loadnet. So this is a shadow fork of mainnet. But in general like these benchmarks can run on any network. And I think it would also be very beneficial for clients to to be able to run these things locally.

So I think it is well I should prioritize or we should prioritize writing these read me so that everyone gets up to date on the on the general status of this this benchmarks and to know what is what's possible. Yeah. It's a good to remind of this. Awesome. Is there anything else on that topic? Any questions regarding benchmarking or something to add?

I haven't been closely following the chat, so if there is anything which that was brought up in the chat um is Um Maybe Maybe regarding the shadow fork. So um it seems that by [clears throat] the end of today we will have Reth also ready uh with all three flags. And uh then we can um think about doing that. It would be great to have all the clients ready. So uh I will think about it waiting for Nethermind to also be ready as well. I will push for that, but I think uh with Reth there is already more of a case to to get the shadow fork the whole thing moving forward.

Awesome, yeah. I agree. This is great news. Perfect. Then it sounds like we are getting very close to getting to the final ballot sign basically. Cool. Anything else someone would like to add before we move on to the next topic EIP-71? Yeah. Um I had a question maybe for for you Yacun. Um on the benchmark releases side we will be uh we'll be using Geth for benchmark releases. Um how does that play into

uh what we're missing for benchmark tests right now? Like those are only going to be able to fill some of the compute tests. Um I realize there's like a GitHub flow for uh benchmarking stateful tests. Uh I myself need a uh need a refresher on how the stateful side of these things work, but um how does the the benchmark releases um through Eels using Geth, how does that play into uh what's missing here?

Okay, so I will also need like a reminder here. Because the benchmark releases, what are what are we talking about here specifically? Yeah. So Eels has uh actual version benchmark releases. And so we use uh we used EVM one to fill these in the past because it's performant. Um I it's my understanding EVM one won't have uh M-chip support, so we've been using Geth. And so maybe we've been working on two separate efforts that I'd like to agree on um and so I I yeah I don't know if this benchmark release is is uh a focus that should be

prioritized right now or if this GitHub uh flow is the one that should be prioritized since this is this is a stateful benchmark which which seems like it's more appropriate. And so yeah, I'm not sure if like if there's like a benchmark release that we should be aiming for or or not. Yeah, I see I see. so the the main uh well the the important point um right now is like okay, do we want like these uh these state locations in the block accesses or not? So that's like the the most uh well the focus point of now also for the benchmarks.

Um but that means that we have to run these benchmarks on top of a stateful network because if you run them on top of an empty network or like a very small network then you will not see like any well no you you you won't really see like a big impact of the block accesses. So what you also need to do is we need to run this on top of like a very big state. So this is why we run this on top of the the block net. for the benchmark releases like uh you mentioned that EVM one is being used for this because it is um it optimized or or or not. But um the well I would say this is also a reminder that I should really or we should really start writing these these weekly, so everyone gets a little bit up

to date on like the these formats. but the part of the sorry what was I going to say? Oh yeah. Yeah. Okay, so so this is this new benchmark format it looks like a payload format like the execution format of the blockchain it tests. and this is like not a benchmark release.

It's like the in-between step between the benchmarking tool and uh sorry between the testing tool and the benchmark tool itself. I think we should maybe just uh discuss this async. Uh because I'm not really sure what the benchmark release was what what you were talking about. Um But the main challenge and the main blocker currently is well converting the tests. And I also see that Eels is up, so maybe you can also Hi. So yeah, I guess the confusing part here is that gas benchmarks when we regenerate the tests are not having kind of official releases. And maybe this is kind of confusing for the others like when when there are new tests being released, but I guess we are

quite dynamic right now and changing a lot of things and in investigating plenty of issues. Still in some scenarios. At some point I want to have gas benchmarks releases with the with the benchmarks which you can just simply pull. Right now everything is like directly injected into the repository which is not ideal, but this is how current workflows were set up and that's why it is not like problematic in terms of uh just doing your work. I mean it it is only problematic in terms of the visibility of when the tests are being regenerated or something like that. So yeah, at some point it will be polished,

but right now we are putting the the the pressure the effort on actually generating those tests and running them to capture the data rather than making it super super clear clean. Yes, okay. That makes sense. And for this uh we have Nethermind set up for for everything that we need, is this correct? Yes. Yes. Yes. Okay. Okay, great. Thank you. That that clears it up very much for me. Thanks. Great.

Anything else regarding the benchmarking? Cool. Seems like everyone is on the same page. I see there is a question from Louis in the chat. yeah, maybe we can answer that one in the chat. Or do you want to bring it up? Uh I I think we could keep the discussion async. Just a quick question for Camilo about the benchmark release from Eels and uh the payload generation from gas benchmarks. I could I could

Yeah. I could ask Yeah, but you you you you described it very accurately. Like the generator is only because we want to generate the tests on top of state existing network. While Eels releases are on top of empty network. But I guess this is being uh planned now and being worked on in in the future to to to for Eels to have this stateful approach as well. Uh okay. Thanks for the response. Perfect.

I see there's there's also been a discussion going on in the chat regarding the what Ben already brought up earlier ordering of the ballot. Is this a topic we want to revisit or should we um stick to what we have right now basically the lexicographic ordering? Um I know that having access-based ordering at least from the specs perspective would add some additional complexity to because right now we're basically doing a two-step two-step process thing where we first collect all the entries in the ballot and then we deduplicate them, which means things that are also changed are removed from the storage reads and so

on. And then we just order them. So this simplifies the whole thing a lot. Of course we could just change the ordering, but for example for reads we don't even collect the transaction indices, which means we would first have to collect the transaction indices in addition to then even know in the end like how to order them. I'm not sure if we if it's worth to reopen that discussion, but I'm curious if there is anyone who wants to chime in. So I I I think if they test surfacing the the data, uh we should just add I mean they already have but pay attention

to the results that we're making. decision on when to cut them. Okay, my point is that if we could infer the access patterns of transactions, what we could do is actually keep the accessed items in the lexicological order, but actually run the transactions in the order of this this this access. So, we don't have to run transactions in in the order they come. With balls, we can do any order.

So, if we can have a heuristic for that, and you know, you can probably do that that just by a sender and like a target, right? Transaction etc. to some extent. So, there might be there might be some cool solutions there to run the transactions in that order. Yeah, I can see that. I can see that. I was I was thinking very similar about this that this is more like individual client optimizations like how to actually parallelize the transactions in the end because of course if we say we want to have it ordered by access,

then we also need to enforce that it's ordered that it's actually put into the ball by access order, which is definitely more complex than just saying okay, it's lexicographically ordered. So, we could have the balls actually ordered lexicographically, but when constructing the balls, we could also add the optimal transaction execution order based on this lexicographical order because when transac when creating transaction you can actually calculate that. It will create the balls you can actually calculate that because when you reorder the transactions, you reorder when you reorder the access patterns, you reorder the transaction

indexes in the same way you reordered those access patterns and you speed up speed out this list and then this is the optimal way of accessing this. Right, but this will then become part of consensus too. So, we will would actually have to enforce it such that this heuristic that you're mentioning that this actually it's actually correct and I'm not sure about the complexity there. It sound relatively easily because basically you have to track when you access addresses, right? And storage slots.

Let's say on addresses for simplicity. You you have to track which transaction indexes access those. And when you'll be reordering, you will get you will reorder you know this those as a tuples and then you get the transaction list order, right? >> Right, but you could run into >> There there is some potentially potential complication that the first thing is accessed only by one transaction and how to call it? Only one thing is accessed early and then a lot of things is accessed by the second transaction.

So, you potentially they could be reordered, but those are like probably diminishing returns. Yeah, exactly. And what I was also thinking of is you could have the first transaction accessing an address, but all the storage keys of that address are accessed in the last transaction. So, you what you then want to do is only load the address or the account and not yet the storage keys that are like nested into that account. Yeah, but then you can reorder the transaction you know, processing order. So, the this last transaction can be processed first.

That's the whole point. Yeah, this is true. I guess I guess we we would just proceed with what Ben said. Let's do the benchmarking and see how far we how well we are doing because in theory if you have like clients if every client loads from disk in the same way for example, every client goes by like lexicographically ordered, then we would introduce this worst case where you basically like in the last transaction you access the lexicographically first addresses and our accounts and then storage keys and so on, but in theory if every client would just start at the random position

and start loading from disk there, then we would enforce that you would end up in this average performance and then there is no attack against okay, I create a block that is like worst case. But we can we can continue that discussion I think I guess and I guess as soon as we have the benchmarks, this will also be helpful to to take a decision there. Can I ask a one question about benchmarks? So, I guess uh it does some kind of small tablet for or even not tablet, but API calls.

I make Note that the stateful tests they are run against the shadow fork of mainnet and bloodnet and everything else that only requires execution can be done against the definite. I see. I just wanted to to mention that balls introduce additional network delay I guess and uh is it counted or not because network is the slowest data like transfer medium rate and then goes the disk and execution is the fastest and uh does it really

benefit us to have balls considering networking overhead in a real network context? Yeah, so with EPBS, there will be more time for execution for block propagation. So, this [clears throat] is definitely one of the things that helps a lot here that we we just have way more time for propagation and then we have 3 seconds at least. That's the current PTC deadline at second nine. You have 3 seconds for execution.

So, we would have to set the gas limit to a point where we can get to the PTC deadline with the worst case block size, but the worst case block size will be a block full of call data. So, we we always have to set the gas limit somewhere such that the worst case block size can be propagated until the deadline. And then if the worst case block size is from call data and the and doesn't contain any execution, you would basically get a block that is right on time at second nine for example and then has no execution. And then we can assume the block the worst case block from a big ball

would basically be somewhere earlier, right? Because it's a little smaller than the worst case block from call data. So, you can assume it arrives before the PTC deadline not right at the PTC deadline and then you have still at least 3 seconds to execute. So, from that perspective, we're good. Of course, we have to make sure that the worst case block size stay a worst case block from from call data and not a worst we don't want to introduce a new worst case block size because this would mean the new biggest possible block would also come with quite a lot of computation.

So, um does it make sense to compare these three cases when we don't use balls but it's balls like don't use balls on any level like consensus on the execution side. Yeah, as the first case and two more cases when we use balls, but in sequential sorry and graphical order.

Does it make sense to compare these three cases or you're confident then that the current approach will be the best? Yeah, I think the access ordering this can be discussed I think regarding if balls are even worth it. I think this is quite clear because if you think of worst case transactions and you have to execute them sequentially versus being able to execute them in parallel, you will just get um the benefits of parallelization. So, this seems quite clear, but the access ordering this can still be discussed if it's if it's worth to to to to improve something on that.

And we since we have the flags to turn on and off the optimizations, we can just compare how we're doing like put the gas limit to 150 and then just run clients with and without the optimizations and see the differences. But of course, we have to only care about the worst cases and for the worst cases, you would basically deal with dependent transactions that cannot be parallelized natively. So, you would only get the parallelization with balls. just to summarize one last thing about my idea.

It would change the worst case worst cases of this lexicological order into more or less the best cases. That's that's the whole idea. So, the best case and worst case in the ordering would be more or less the same. So, we can So, if there is a big difference with best and worst cases, we can revisit that idea. But, on the other hand, I think we might even just drop the whole whole access parts with the person.

>> State locations, yeah. >> It might be Yeah, state locations. So, it might be So, it might be worthless. But, the whole thing that to summarize it, it would make the worst and base case of state locations more or less the same. So, that's that would be the benefit. Yeah, I remember getting getting from Gavin some benchmarks early on just to determine how much we get from batch IO. And back by the time I remember he got a 10x to 20x improvement if I remember correctly just by employing batch reading instead of sequential reading.

But, we can we can determine that with benchmarks. Right, we only have 1 minute left now, which is not a problem. I just wanted to give a heads-up regarding snap two. it's EAP 815 8189. Basically, this EAP was merged today. I would ask clients to have a look at it. I know that almost all clients except Aragon will probably want to make use of it.

What it does is it's a quite significant improvement for the healing phase. So, instead of using get try notes, you would just apply the ball. Um it's even a quite significant simple simplification for the healing phase. And yeah, this is more like a heads-up. It's not needed for definite three yet or um something like that. just for clients to have a look. Perfect. We are at time. Is there any last comment?

Perfect. Thanks for posting the EAP. And then, I would say we are done. And thanks, everyone.