Block-Level Access Lists Breakout #6

Transcript

Toni Wahrstätter: Hello, everyone.

Karim Taam: Hello?

Toni Wahrstätter: Let's give people one or two more minutes, and then we start.

Okay, I think we're ready to go. Perfect. Hello, everyone, and welcome to the sixth breakout call of EAP7928. It's November the 5th. And let me directly post the agenda into the chat. So that everyone can have a look. We have quite a few topics, because I put all the spec changes we made in the… since the last breakout call into… Into the agenda today, just so that people are aware of what happened since last time. This basically includes fixing how we track withdrawals, so you might remember in the last breakout call, we discussed how withdrawals are handled, and we had some bugs there in the specs.

Besides… Besides that… Yeah, there weren't… any other major changes, at least as far as I'm aware. Maybe, Felipe or Rahul If you want to add something to that, feel free.

raxhvl: I think that covers it, Tony.

Toni Wahrstätter: Perfect. Awesome. Yeah, have a look at the agenda. I linked all the specs, and besides that, you can see the agenda point proposed changes, and this is then directly the first topic that we should discuss today. Namely, do we want to include gas used values of each transaction? in the block lab access list. And… I've already spec'd it, so you can look at the specs, how it would look like. And the main motivation behind this change was allowing clients to better parallelize transactions and avoiding Worst cases that originate from not being able to distribute transactions. Evenly across your course. So, I'm not even sure if this is the right, mental model I have on that one, so I'm curious what client teams think about this one.

If the gas used is actually something that would be helpful, or… If you think, like. Just naively, parallelizing them in the sequence, how they are in the block. Is… is good enough, or… Might even, yield the same results. So if anyone has thoughts on this one… I would just… I would assume that the default is just we don't do it. But I think we should consider it, at least thinking of, Certain worst-case scenarios where… You could end up with, with big transactions, being executed on one core, while other cores might idle.

Gary Rong: So, personalizing it's really hard to say that how much performance gain we can have by adding these, inductor. One good way is we can try to compare the So, execution time. With this, gas used, indicated or without. So that, We can, like, try to measure the real, blog, and, then we can decide if we want to include it or not.

Toni Wahrstätter: Yeah, good point, and maybe just one question I would have regarding that. What would actually change if I give you a list of the gas used values? Would you just, Modify the order in which you present transactions to the exec… to the executor. Or is there a process where you're really like, okay, I have transaction 3, 5, and 7, and they are all very big, so I make sure I distribute them across core 1, 2, and 3, and make sure they are not landing on the same core? Is there even something like this happening?

Gary Rong: I hadn't really, like, thought about it, but I guess, We will try to, schedule the heavy transaction first, so that they can be… there's no waiting time, and we can try to… Give it, like, more execution time. Yeah, it sounds like a very naive approach, like, we schedule the heavy transition first, and then… try to emit the transaction one by one to different threads. Yeah, the… I don't have, like… Any… like, a useful comment for it, because I never… I, like, haven't did any experiment about this.

Toni Wahrstätter: Yeah, Dragon?

Dragan Rakita: I think it's not worth it. Rough math around that, the worst case, if… If the stealing of the, like, scheduling of transactions to the course is done correctly. The worst case can potentially be… because we have, like, 60 million cap limits, 60 million gas per transaction. The difference is, like, 50% faster execution when everything is parallel. With both, that the speedup is, like, 3-4 times, 50% doesn't matter at all. So it doesn't, if the workstealing is done correctly, I don't think it's worth adding that.

Toni Wahrstätter: Yeah, I see the SEO point. Yeah, the example, a more tangible example, might be assuming 60 million guests, and we have, like, or a little bit above 60 million, we have 4. let's say we have one max size transaction, and all the available… the remaining gas is consumed by EVE transfers, for example. What you would want to avoid is executing all the EVE transfers. before the big transaction. So what you want to do is parabolizing them, I guess? Yeah, but I can see your argument that it might just not be worth it, and just confirming also what Gary said. Go ahead.

Dragan Rakita: In… in general, at least, I still didn't… I'm not sure even if I'm going to implement that, but you want to implement something like steel intersection. You have 4 threads, or 4 cores, that execute in the… in the parallel, and they steal for one… one single queue. And whatever the core finish, executed transaction, it steals another transaction from the queue. In that sense, every core that's not… that wants more work will steal the task from that queue. At least, this is, like, one of examples how could this, like, work. Worst case for that, if the last Task is the biggest one. And the biggest task is 60 million, so it's like… It's… yeah. It's not worth it, to be honest.

Toni Wahrstätter: So compared to, like, 60 million gas limit, it's already, like, one-fourth. And compared to, like, 100 million, like, the effect will just become… Smaller and smaller.

Dragan Rakita: Yeah, exactly.

Toni Wahrstätter: Yeah, anyone else who wants to chime in, voice an opinion? Otherwise, We don't have to take a decision today anyways. The default would be just we don't change anything, and the gas used would stay out of the block of access list. We can also use, the time around DevConnect to discuss that further. Maybe clients, get to comparing how they would eventually parallelize But in general, I agree. This argument of Dragon does make a lot of sense, that We might improve the worst case. But it might not be worth it.

Po: hello, Tani. Yeah, go ahead. I have a question. Because I joined later, the question is, shall we schedule the transaction execution by guest used?

Toni Wahrstätter: Yes, exactly. If we put the gas used values into the block lab access list, if this is kind of useful information for parallelizing.

Po: Okay, I recently just did an experiment with gas limit. Yeah, it seems this information is Enough, and it seems the most time-consuming Transactions will dominate the block execution time. Yeah, maybe later I can show a demo, so you can see the results. Yeah, we can't schedule just by the gas limit, yeah.

Toni Wahrstätter: Exactly, yeah, this… these are, like, the… there are three options, scheduling just by… or four options. Just naively scheduling, by their order, as they are in the block, then you can use the guest limit. then you can use the gas used, which would require us to add it into the block of access list, and the most hacky solution would be you take the balanced diff, and from that you infer the gas used, because Every, every time you pay fees, it's also a balanced diff. yeah, it might be attackable, and the cleanest solution would be to just use the gas used, but as Dragon said, Worst case scenarios are still, like, very contained.

Mark Holt: Bye.

Toni Wahrstätter: the 7825 transaction cost limit. And we might just ignore it, ignore the whole topic, and don't include it. But yeah, definitely happy to look at your results there.

Mark Holt: Yeah, so sorry, just, I joined a bit late, so I'm not… So, one of the things that we've discovered with parallelization in Aragon is we've basically ended up with two phases. We do the execution phase. And then we do the receipt phase at the end, because… Depending on the order in which parallel execution is happening, if you use a common gas port, it basically, you're gonna go out of gas. Essentially, when, in fact, you wouldn't, once you… once you get to the end of it. So having… having the gas per execution would make that process a lot easier, basically. Because it means while you're execute… you know, when you execute a particular transaction, you can check its gas at the time.

Which you generally can't do when you're executing in parallel. Because you don't know the order. If you see what I mean.

Toni Wahrstätter: Yep. Yeah, this is… this is… this is exactly the argument. Yeah.

Mark Holt: I had too, yeah.

Toni Wahrstätter: Dragon's counterargument is, like, that even if it turns out you scheduled them in a way where you did, like, the worst job possible, for example, you were tricked into scheduling that way, then the worst case would still be, like, you would maximally have, like, one transaction on top. While all cores are idling. And this transaction only has a size of 16 million, which is, like, 1 fourth of… Yeah, what do you mean?

Mark Holt: The problem is more that if you've got transactions with reverts in the middle of them. As you're processing, the transaction looks like it's using more gas than it should do, and… What then happens is, it's generally fine, but if you're on a full block. And you get to the final transactions, you can basically run out of gas too soon. If you see what I mean. So you get a set of edge cases around… Failing blocks, on gas. When actually they don't, because the… effectively, the transaction looks like it's using a lot of gas, then it reverts. So…

Dragan Rakita: I'm not exactly following this use case. Are we talking about if the intersection is not spending exact amount of gas that it should be? Or something else.

Mark Holt: Well, I mean, the transient gas cost of a transaction Changes, depending on whether it reverts or not. And… Basically, that means you end up with some cases when it looks like they're overusing gas. So you get out of gas sooner than you expect. If you know what the transaction gas is. You're never gonna hit that situation.

Dragan Rakita: You mean if the trace section reverts, all the gas are… He spent, even if the gas is not, like… used. It's it like that.

Mark Holt: Yeah, it looks like it's… so it depends, to some extent… extent, it depends how you're doing the gas calculation.

Dragan Rakita: Yeah, then this is, like… Good case where the cycle should take a lot more time, but in reality, it takes one end of the time than it should be. It's like, yeah, it's faster to execute than we expect from the gas. Okay, that's a good comment.

Mark Holt: Yeah.

Jared Wasinger: Why is it's not clear…

Mark Holt: Does this…

Jared Wasinger: Sorry, go ahead.

Mark Holt: It just executes differently, and basically what we've discovered in testing is you get the occasional block. that you go out of gas right close to the end, basically. And it's only… so it only impacts full blocks, basically. Or foolish blocks.

Jared Wasinger: Yeah, I… I don't fully follow, but I… just to be clear, like, the implication of this… Of not having the gas used here is basically that if we… That we can't invalidate a block for the actual gas use of all the transactions, going over what is allowed by the gas pool. We can't do that until we, like, we have to execute all the transactions and validate it at the end, right?

Mark Holt: Yeah. In the collection of the receipts? Is that… that's the issue you're getting at here? Yeah, yeah, that's the issue.

Jared Wasinger: Okay.

Toni Wahrstätter: Let's see. So, essentially, at the very end, you would go through all the transactions again, and determine if they actually Fit into the block.

Mark Holt: Yeah, which, you know, in practice, it's not a very computationally heavy process, so it doesn't make much difference to the… because… because you need… the thing is, you need to do that for the receipts anyway, because you've got a… you need the… you need the aggregate number as the… so you end up having to total the receipts at the end anyway. And comparatively, it's very little processing to do that, so you don't get a big advantage in doing that. Well, doing that whole process at the end doesn't seem to cost much time, in terms of the scheme of processing a block.

Toni Wahrstätter: Right, and then…

Jared Wasinger: The main downside is… is just that you can't… you can't… You can't know that the block is bad until you've processed all the… Yeah, correct. transactions in this circumstance, so…

Toni Wahrstätter: Right, but this is not really a DOS vector or anything, so… to me, it feels like… This is not really a strong argument for having the gas used values in the popular excess list. I do see why it makes this process cleaner. Because you could just compare each transaction individually and independently to the gas used value in the popular access list.

Mark Holt: Yeah.

Toni Wahrstätter: Yeah, so it's a nice… Yeah, good point.

Mark Holt: I'm just… yeah, so, in the scheme of things, it's a nice path.

Toni Wahrstätter: Right, yeah. Yeah, I agree to that. Yeah, I think… I think for now, We would… we would not include it, but, yeah, we still have some time to recite this topic. We don't have to take a decision of, Today, So, we can also discuss, during DevConnect and at the next, breakout call again. Yeah, it's good that we have, Brought this topic up so that people are aware of this possibility. Any more comments on that? Perfect, then let's move on to the next agenda item. Actually, the next agenda item is client, indefinite updates.

So, if you want, Stefan, I would invite you to Give us an update there.

Stefan Starflinger: So, we, overhauled a little bit how we do definites. We added a new boot node, setup, so that's why we've been kind of iterating on our side a little bit. I can also share, kind of, the spec updates, that I haven't mentioned yet. PK has, added, EVM fuzz. Tool to the spammer that we can use. It basically creates contracts with, different opcodes. And if you run that with different client pairings, there's still a few issues that can be found. So I'm generally kind of waiting for a very or relatively stable kurtosis testnet. And to see that all clients are decently working together. And I think also with the spec changes, it would be good to get some client updates also.

kind of where we are at, because it's, it's, wouldn't make sense to spend money on a DevNet, if it's not running stable yet.

Toni Wahrstätter: Yeah, thanks for the update. Maybe let's ask client teams how… Far in the process are we, or how close do you feel being able to turn a definite? When it would make sense, of course.

Jared Wasinger: Yeah, I guess I'll just go, I think that Geth is mostly ready, but, like, so we just had this new… test release… so, essentially, the Coinbase issue hasn't been entirely settled in Geth, But other than that, we would be ready to go.

Karim Taam: On Bizzou, we are passing all of the tests related to Amsterdam, but we have a… I think, as other clients, we have a lot of issue in the tests that are testing old fork. So, I'm not sure if we want to be sure that we are passing this test before starting the DevNet, or if we just say that it's bad configuration of the test, and we will see, maybe in the next version, if it's okay. Because we don't really know if it's just because of a bad configuration in the test, or if it's really a… A real issue or something, isn't it?

Stefan Starflinger: Do we have someone? Yeah, sorry.

Marc Harvey-Hill: I was just gonna say, for Nevermind, it's the same, just, going through these new tests, This new test release, and just checking, why some of these tests are failing.

Dragan Rakita: Same with RAT. I don't have any pending items on both sides. So… Fixing tests on spec side, running them to confirm that everything is okay. It's, like, the next thing. And after that, I think DevNet Zero could be spun up.

Mark Holt: So, where we are from an Aragon perspective is we've now got an implementation that we're testing. We're getting quite a few Hive test fails, so… Chome, who's actually working on it, is not on the call, because it's not good from a time zone perspective, but I'm wondering if there's somebody he can contact, just to find out whether Because this is the first time we're going through, he's trying to work out whether the fails are… us, or DevNet issues, testnet issues, or Hive issues. So, it would be good if there's somebody that we can put him in touch with. Because we're kind of bootstrapped in the hive process at the moment.

Toni Wahrstätter: Yeah, definitely. So, I think the… the two people with the most insight on that process are Felipe and Rahul. So, yeah, Rahul, I just see you on the call of the deep end, maybe you could get them a little bit?

Felipe Selmo: Yeah, yeah, I'd like to catch up. With everyone on the state of the tests, so… There are a few spec changes that are needed that, actually impact… So this is based on conversations on the block access list, Ether and D channel. There's a more recent conversation with… with Sam, who's… Been leading the… the Execution Specs project. And these changes are going to actually impact the specifications for all forks. And so what I want to do right now is release a patch. For the tests that has The changes that we need so that we can Generate cleaner tests for block access lists. And this includes all the old forks. So what this release did is it basically

Build all of the tests that we have in our test repo. Across all of the forks. generating block access lists for all of them, and so I expected quite a few failures. Because this will reveal edge cases that we just haven't written targeted tests for block access lists yet. But knowing that there are spec changes that are needed, means… Right now, it's really hard to tell, which tests Are, actually bad tests, and which ones are failing because the specs aren't up to date. So there's this… Discrepancy of resolving the specifications quick enough for Amsterdam, at least, so that we can Put out a patch release for the tests, but also, ultimately, we have to go back and

Fix the specifications. It's basically related around charging gas. For, memory expansion and… Loading account earlier, within each call. And then continuing on with the call afterwards, so that we… Only track the account if we have enough gas to track the account, and things like these. And so, yeah, unfortunately, the current state of the tests don't… aren't very representative of What the specifications, should be. Currently.

Toni Wahrstätter: Thank you very much for the update. Yeah, I've just posted the PR I'm working on in the chat. Just to summarize. there are two things… some things are more like cleaning up. For example, putting the state tracker into the block environment instead of the state. The more crucial thing you were referring to is really Kind of splitting up the charging gas from checking the gas, and making sure that if we Run out of gas. or let me put it differently, that we never put an address into the block club access list, or any entry into the access list. When we run out of gas afterwards. Right, so you want to kinda… what, Dragon… I think Dragon was that… was saying that, or…

who was… I don't remember who brought it up, but is that already, like, addressed in my PR?

Felipe Selmo: I… I'll need to go back. This is the new PR, right, after we've had conversations with Sam?

Toni Wahrstätter: Right, yeah.

Felipe Selmo: Okay, yeah, I had the… so I also posted my gas charge split PR, and this does, stabilize all the bad tests. And this is… I'm not sure if this is all of the changes that we need, but I think this covers most of them. And it does have all of the targeted block access list tests that are up to spec passing, and so I feel like this PR that I worked on, and whether those changes are also in your PR, some version of that. Is what we need for a more stable test release.

Toni Wahrstätter: Perfect, yeah, then I will take those changes out of my PR and just leave it at this refactoring, where we move the, state tracker around.

Felipe Selmo: Yeah, that sounds great. Internally, I think I just… I would like to get, The steel team on board with putting these changes up for Amsterdam first, and then we can focus on the full forks, refactor to how the specifications charge gas in general. After the fact, so that we can… Get our tests up to speed.

Toni Wahrstätter: Perfect, thank you. Awesome, yeah, I haven't, haven't seen your PI yet. I will, I will review it. Perfect. Any other, updates we should discuss? Regarding testing, regarding the specs? Making sure we don't forget anything. I'm relaying this message from Stefan. The clients please run the EVM fast scenario for Spammer. Great. If there's nothing else, then we can move on to the next agenda item. Which is… A rather old topic, it's still about, do we want to include the reads, or… Don't.

With reads, we usually mean the state locations, so unmodified accounts that come with empty, lists of changes objects, and also storage keys? So everything that is touched within the block but not modified. We have discussed that topic, a while ago. And Gary did some benchmarks very recently. Gary, are you on the call? Do you wanna… Quickly.

Gary Rong: Yo.

Toni Wahrstätter: Yeah, what you did there.

Gary Rong: Yeah, I can, share my… right up, can I share my screen?

Toni Wahrstätter: Yeah, should work.

Gary Rong: Okay, can you see my screen?

Toni Wahrstätter: I can see it, yeah.

Gary Rong: Yeah, so, I've been discussing with Tony recently about whether to include the read location in the BAL, So, at the beginning, I was at the opposite side, because I think it's not that easy to… properly gets all the relocation during the block's execution. And, because we need to compare the BAL data in the consistency, So if we provide, like, slightly different data, since the consensus will be few. like, I am not very confident about it, so I suggest that maybe we can, Exclude the location in the initial version. But, yeah, later I try to, measure the performance gain we can have by including… by utilizing this relocation.

So, I think the intention of the BAI is to try to optimize the block's execute… block execution, and, so that we can push the network S limit, upwards safely. But we need to ensure that the BL can benefit, basically, all the potential scenarios in the block's execution. One potential worst case I can imagine is that the read-heavy block. And so now the… gas cost for the code as load is. $2,100, and for the account access, it's 2600. So, with the current network guest limit, we can, in theory, have, 20… thousand, S-loading within a single block. Yeah, so I try to measure, like, how much time we need to spend if we want to access this

So many, storage or accounts. And, with Osaka, like, we limit the maximum gas limit within a single transaction to 16 million. So, It means that we can at least have 4 transactions within a single block. So the baseline I have chose is a thread with 4, So the first thing I did is, I tried to measure the batch rate with different thread over the raw SSD. And, yeah, it turns out that, with 34, we can achieve this, 70s, 7… 70,000 QPS, curry per second, and with 128, we can achieve, Roughly… 10x the speed up. so, yeah. And,

like, it's the real performance over the raw SSD, and the next thing I did is I tried to perform, this batch read over a real E3 million guest node. The background is that, the database size is, 288 gigabytes for the… Key-value store. And, yeah, so… With 4thread, so… QPS is around… Mmm, 22K. And, if we give, like, 128 thread, the QPS can go up to, 200K, so it's 10x faster. And this number, like, totally changed my mind, and I think it's worthwhile, because, It is 10x faster for this particular, rehabic scenario.

And another interesting thing is, I've… I have observed is that, before the thread, 32, if we give more thread, like, higher concurrency, the reperformance, will be increased linearly, but after that, we, like. Hit the acceleration point, and the performance… there is no significant performance gain by having more concurrency. So I think it might be relevant with the physical structure of the SSD, because internally it has some… Channel inside, and it has a specific capacity. Yeah, so let's say the acceleration point is 32 thread. And, given that, for a single transaction, the maximum gas limit is 16 million gas. So, I think we can say that

Before the network gas limit, be increased to 500 mega… a million gas. So BAL waste red location will still be beneficial for us. Yeah, but all the benchmarks are performed over the gas implementation, like, different client can… because different clients use totally different database, and the number might be slightly different, or could be completely different, so I would recommend you guys to simulated the same thing, and to make sure, like, our numbers are aligned with each other, so that we can make, so… Final decision to include it or not. Yeah, I guess that's it. I did link my benchmark program here, so if you want to have a reference implementation, you can check it out.

Yeah, but otherwise, syncs, that's it.

Toni Wahrstätter: Awesome, thank you very much. Are there any questions people have?

Łukasz Rozmej: So, one comment from me, because Gary used 28 thread CPU, so this 32… thread limit, increases might be as well from the CPU being, completely used, but I don't know. That might be a good thing to, check. So, maybe on different CPU and different disk, or, like, especially, like, disk in RAID, we could achieve even more, scaling.

Gary Rong: Yep. It could be possible, so, yeah, we can try to use a different setup to run the benchmark again, and .

Mark Holt: Sorry, so how many CPU threads… so how many CPU threads have you got? Cause you, my…

Gary Rong: In my setup, I use, Intro, like, 147… K. So it should have, like, roughly 20s. cost for… concurrency.

Mark Holt: Yeah, so the question…

Łukasz Rozmej: 20 cars and 28 threads.

Gary Rong: Hmm.

Mark Holt: But, so, for Disgao. you're not… it may be offloaded off the core. The interesting question is, if you have fewer cores and the same number of threads, what does it do to the performance?

Gary Rong: Yeah, I think, When I tried… when I did my benchmark, like, it is the only program I am running, so I would assume that it can use all the, available cores. But, as I said, in my setup, like, I use this particular CPU and this particular SSD. If we use something different, maybe the number could be, like, slightly different. So, yeah. Yeah, I'm just wondering how much CPUs were doing.

Mark Holt: Because they're basically. this guy, or… or device I have.

Gary Rong: So in this, benchmark case, I… basically, we have no, CPU, computation.

Mark Holt: Because, we assume it's a really heavy…

Gary Rong: block, and the read is the slowest operation. So, for the worst case, it's just all the read without any VMIC computation.

Mark Holt: Yes, that's what I'm saying. So that implies that, actually. you can… your number of threads is not CPU dependent. It's… you should be able to increase the number of threads you've got allocated to I.O. Because… the CPUs are just idle. They're probably waiting for the I.O. to return. Because it's the I.O. subsystem that you're loading. If you see what I mean. Anyway, so it'd be interesting to see what the numbers would look like on less threads. I mean, less CPUs, same number of threads.

Gary Rong: Yeah.

Toni Wahrstätter: Yeah, and it would be great if clients can also reproduce the… reproduce that, individually, just if we have. more data points on the topic, as well as we should try it with double the state size. So currently, in Gary's analysis, the state was at 288 gigabytes. I think right now… We have a state growth of, like, 60… 6200 gigabytes per year. So, we should definitely try it with double the state size, too. And see if this changes something in the numbers. But in general, because one thing we have to consider when it comes to the state breeds is that the worst case Block level access this side. Size is also for the exact same scenario.

So, we might end up in a situation where we have, like, the worst-case block containing a lot of storage reads, while a rather big block… it's not the worst-case block, because call data will still be Allowing you to get to a bigger block, but it's a… it's a big block. So, for example, at 60 million guests, the blocks of access list would be at, like. 850 kilobytes to 1MB. Which takes a… with 50 ambit, takes 165 milliseconds to download. So this is all something, all things we have to consider here. So if we, like, get something from the state locations, then it must be significant, and it must be Worth the costs we pay, on bandwidth.

But based on this analysis, it looks like it.

Mark Holt: And it looks like it's… it's…

Toni Wahrstätter: tops at 500 million. So, like, at 500 million, we have enough transactions That you can, nicely parallelize them. And then… The benefits we get are a little lower, or less significant. And… Yeah. Go ahead, Mark. You wanna say something?

Mark Holt: I was going to say, the other… the other thing, though, is that if you're running… when you start running parallel execution, if you parallelize the transactions. You get a fair amount of natural Parallelized reads, anyway. Right, because if you've got all… if you've got all your executions running. N of them hit a read, so you're running reads in parallel anyway. If you see what I mean.

Toni Wahrstätter: Exactly, exactly. That's why Gary took 4 as the benchmark, because essentially, with 60 million transactions. You will be able to parallelize 4 transactions at the same time. Which is essentially the same as doing, like, Using 4 threads for reading. Right? This is what you're referring to.

Mark Holt: Oh, yeah, and I'm saying, basically, so, if you've got all the… I mean, in practice, if you've got all the reads up front when you start parallel execution, you'll kick off a set of parallel reads, but also, you've got, like, a race condition in the sense that you won't… I don't think you'll wait to start executing transactions, and then, kind of, the early transactions will actually go to the disks themselves. Right? Because they get there before the read comes back. And the question, if you care about bandwidth, is what boost do you get over the natural parallel flow, if you see what I mean? Because you could be driving the disk quite fast anyway, just from the parallel execution process.

So you might not need… you might not need the read list, because the transactions will get there pretty quickly anyway, if you see what I mean.

Toni Wahrstätter: Yeah. Yeah, that's a good point, and that's, I would say, exactly the reason why we have to consider if reads actually give us enough to be worth the costs we pay. Right now, based off Gary's analysis, it looks like it is worth it, because It was, like, 8x or something, and in absolute numbers, a few seconds. Difference when we are at 500 milliseconds? Or a few, tenth of a second difference? if I remember correctly, like, a little more than half a second of difference, and then it looks like it's worth the bandwidth costs we pay. But yeah, I think… I think it's very close, and we should definitely get some numbers on Different clients, different implementations of the simulations, and second, on a larger state.

And, yeah, maybe we can then take a… take a decision. As soon as we hefted. I just see Dragon in the chat. There is EAP7870, if I remember correctly. That defines, like, minimum hardware specs, ELP. 7870, am I correct? Yes. I will post it into the chat.

Łukasz Rozmej: Can we just implement both and run real-world, like, testing later, and decide closer to the deadline? on that.

Toni Wahrstätter: If you're happy.

Łukasz Rozmej: Being measuring both, right, having both access lists with and without, or, like, maybe distribute with, but then measure execution with and without, right? On one machine like that, one machine like that, and compare The actual results in practical blocks, and worst-case blocks, too.

Gary Rong: Yeah, it is, like, the intention of makes this benchmark, because we can… Like, have a… Feelings at how much performance difference we will have. So, Like, if we run the BAL in the real, scenario, it would be perfect, but at this stage, the early stage, we can try to measure the performance difference with this benchmark first. It's what I'm trying to say.

Mark Holt: Yeah. But I kind of agree that we need to go through… we should leave it… so I think leaving it in at the moment is a good idea, and they… we ought to leave it in for quite a long way through the… the DevNet process, because once we've established some operating parameters, what we actually want to do is take it out and see whether it makes any difference. If you see what I mean. Because otherwise, you really don't get the… the… what does it do to the overall end-to-end parallel flow? Because one of the things that I've discovered testing parallelization in Aragon is, you really need to have all the pieces together. Because they… they have different behavior when you test them individually.

Basically, in the end.

Łukasz Rozmej: So, if we have them in for now, we don't even have to take them out, we just have to have a second implementation that doesn't use it, and we can compare it live, the difference on…

Mark Holt: Yeah, yeah, correct, yeah, yeah. Yeah, I agree with that. Because you can say, well, what practical difference does this have over your block time if you use it or you don't?

Toni Wahrstätter: Yeah, I can see that argument. The only thing I was, Hesitant about was taking a late decision on that, because, of course, we would love to… to have the numbers today instead of tomorrow.

Mark Holt: Yeah.

Toni Wahrstätter: So I don't wanna… you know, I'm not sure if it's the cleanest approach to have, like, two releases running at the same time, or if we… Just do the benchmarks now, and make sure we get it… we get to as close, as realistic for a south as possible. I mean, from the spec side, it wouldn't be a big problem, but I would assume that in client implementations, it's not just taking them out versus leaving them in.

Łukasz Rozmej: Not really, it should be relatively easy to do, because you could just have a parameter if you load those, in the background thread, based on this access list, or you don't, or you just ignore the street access list, right? In a way.

Toni Wahrstätter: Yeah.

Łukasz Rozmej: So, I…

Toni Wahrstätter: So there are some edge cases attached, right? For example, if you revert and you have written to something in the specs, for example, we have this custom function that converts a write to a read for reverted writes. stuff like this would then be needed to be commented out, but I agree, it should be possible to… Yeah. To do so.

Gary Rong: Yeah, and another point is that Personally, I think there are so many HGKs in, like, collecting the relocation. If we can somehow make an early decision that we don't want this relocation, it would be much easier for the implementation.

Toni Wahrstätter: Right, yeah. Yeah, the good thing is that we have… we have them in the EAP right now, and clients are basically ready to… to ship them, or on the track to ship them. Compared to adding them later on, which might be… Way too difficult. So, I… I would suggest we keep them in for now, but it's great that we now have more data points on that topic. And then it would be great if we can simulate it even more. Maybe we get some clarity before we even need to run multiple versions on a defnet and compare them empirically. But yeah, let's see. Let's keep the discussion on that, ongoing. And keep the thread open. Is there… Any… anything else we should discuss regarding, this optimization.

Mark Holt: No, I just want… I mean, it's… the summary of the discussion seems to be that benchmarks doesn't rule it out, right? It says, actually. is potentially going to give a big enough performance improvement to… that we should continue with it. That was what I was hearing. Gary's saying, I think.

Toni Wahrstätter: Yeah, and Gary's benchmarks were also, like. not super clear on what, what is the actual, to-do from them, right? Because they give us, like, an 8 to 10X, With diminishing value, with increasing, block gas limit? But at the same time, ignoring, increasing state. So, I think… I think it's very difficult to… To tell if it's worth it or not, just based on that. But it's definitely a first data point we can anchor, anchor on.

Łukasz Rozmej: Interesting conclusion for implementation is that if we do have read access lists, we probably want to spawn, as many threads for transaction processing up to the num… we have on our CPU, but if we don't have read access lists, we probably want more threads on transaction processing than we have it will be stalled. So that's an interesting point for the implementation.

Mark Holt: Yeah, well, I think the reason I was asking about how much CPU has it got, the problem is, if you dedicate CPU to reading, you're not dedicating CPU to, computing transactions. So there's going to be a trade-off there, potentially.

Gary Rong: Yeah, but, I mean, like, in this particular scenario…

Toni Wahrstätter: Same for the block… for the post-stage route, I guess, right?

Mark Holt: Yeah.

Toni Wahrstätter: Perfect, yeah, thanks a lot, Gary, for the benchmarks. This is super helpful.

Jared Wasinger: Gary? Yeah, go ahead. Gary, you were starting to say something, and I…

Gary Rong: Can you hear me?

Jared Wasinger: Yep. Yeah, I can hear you.

Gary Rong: Totally. So what I'm trying to say is that, if we think that the read is, underpriced. So the worst case will be a block with all the, as load, with basically no EVM computation. So in this case, if we dedicate all the CPU cores to the S load, it should be fair. Because there is no, other EVM, computation, and, EVM computation is, like, Way more faster than the discrete.

Toni Wahrstätter: Is that… is that realistic? Is it even possible to do, like, only S-loads without any computation in between?

Gary Rong: I'm not so sure, but I guess, it might be feasible to have some specific program to to do the, for example, external code hash for a non-existent account, and this address could be generated, like, randomly. In this particular scenario, it will basically touch all the pieces of the database, so we can avoid the data locality in the cache. It should be possible to correct such block to slow down the block execution, even with the current mainnet.

Toni Wahrstätter: Thank you very much, yeah. Is there any… anything else on that topic? Otherwise, let's get to the last point in the agenda, namely, in person. meet up at DevConnect, so we have space at DevConnect that we can meet up and discuss, everything around block access list, progress. There is no… Daytime yet, so this is still something, I will work on, but I will keep… everyone updated, and also make sure that people that are not at DevConnect will be able to join via Zoom, or… We can meet Torin, or something. But yeah, just as a heads up, we might meet and discuss things like the gas limit, the reads, and so on, at DevConnect. Perfect! Before we end the call, is there any open topic we should discuss? Anything else we forgot that is, like, important until the next breakout call in two weeks?

Felipe Selmo: Just on what…

Stefan Starflinger: Sorry, just on what Philippe said, there are, like, the debugging, there's just the hash if there's a mismatch between the bars and the logs. But there's a debug endpoint that Bezos implemented, maybe other clients might want to look into implementing that, where they have the bad block that they disagree with, and they have the access list that they generated, so it's easy to compare.

Toni Wahrstätter: Good point, yeah. Thanks, Stefan, for bringing that up.

Felipe Selmo: Yeah, that's a nice second step, too, though. I feel like because we have so many hive failures, if we had this in the debug logs, it would be… Quite quick to just have them to compare against each other. Something Mario, from Steel brought up is… It would… it would be good for a future like this if we had, tracing aligned, too, at some point. I mean, this is somewhat in the future, but how we do EVM tracing, but we can trace what the bow is doing for each client. But… this debug, debug log of the full valve, on errors would be… would be quite nice, I think.

Mark Holt: Yeah, having spent the last… several months debugging parallel flows, that would be definitely useful, because you end up needing… what you want is one that worked and one that didn't, and to do a compare of them. Otherwise, it's really difficult to work out what the hell's going on.

Felipe Selmo: Yeah, exactly.

Mark Holt: Yeah, so this endpoint, does it have a… has it got a specification? Is there something we can copy so we don't do something similar but different? The basic one you mentioned.

Stefan Starflinger: I don't think it has a specification. It basically just dumps the block as a top-level key, and under it, it's the generated block access list as another key, and that's pretty much it.

Mark Holt: Right, so it's basically just the block, and so, in the response, it's the block… And the bow, basically.

Stefan Starflinger: basically, I can share, in the channel, in the research channel, I can share an example as well.

Mark Holt: Yeah, no, that would be great, just so that we don't… because we might as well just copy it, so we're producing the same thing.

Stefan Starflinger: Yeah, and in the spec I shared above, I also shared the curl request to get it. I'll also share an example there, yeah.

Mark Holt: Yeah, no, that would be brilliant, just so that… because we'll definitely implement it, because I think we need it, basically.

Stefan Starflinger: Sounds good.

Mark Holt: Yep.

Toni Wahrstätter: Perfect. Awesome. Then we're at the end of the call anyway. Thanks, everyone, for attending, and see you at the next one in two weeks.

Mark Holt: Okay.

Felipe Selmo: Oh, thanks.

Gary Rong: U.S.

Jared Wasinger: Yeah, bail.

Transcript adapted from ethereum/pm, CC BY-SA 3.0.