Ethlambda Experiments

Transcript

Um hello. I'm Mega. Um, in this talk I'll uh, tell you about the experiments we've been running in the as part of the Lincoln Census effort. Uh, >> [sighs] >> Um, to start I'll explain what I mean by that. First of all, if Lambda is Lincoln Census client made by Lambda Glass, uh, we've been running Lincoln Census interrupt devnets and we recently also added support for the beacon chain and we're currently staking on the Goerli and Sepolia.

Now, for the for the experiments, uh, the goal of these experiments was to is to find the limits of the current architecture of Lincoln Census and find ways to scale it. Be it by having higher validator counts, lower time to finality, or lower slot times. Um, and to give you a bit of context, for anyone that doesn't isn't isn't part of the Lincoln Census stuff.

The current Lincoln Census consensus algorithm we are using is modified is a modified version of LMD Ghost with 3 S of mini. We have some documentation explaining it on this QR. And the minimum time to finality of this configuration is three slots like the three three S of name says. Uh, which is our target of optimization. Basically, if we reach three slots we know the the network is is healthy, basically.

Um, so Lincoln Census, the version we use for this experiment is PQ devnet five, which is the latest one that was released. And part of this Um, of this spec is the slot configuration, which uh uses 4 second slots divided in five intervals. Uh, but basically what each slot uh what happens in each slot is that validators sign votes. Aggregators aggregate those votes and propagate aggregated payloads. And the the proposers the proposer of each slot then packs these aggregates into a block

proof and um propagates it. Uh, you can see here in this diagram. Um, finally uh the last bit of context. We are using Limbi um the one I think everyone here knows about it. Um, it previously used consensus uh it used Poseidon as everyone knows, I think. Uh, we started doing this experiment with that version of of Limbi um we later updated multiple times and we it's always uh now using Black 2S, I think as a placeholder it was.

Um, and yeah, uh now that was the last bit of context. Uh, we'll start with the experiments now. The first one we run was a simple kind of smoke test a 512 validator devnet. Uh, this one we run on a single server with eight cores and 64 GB of RAM. Um, the con- the devnet was basically 32 nodes in the single single server. We used uh four subnets uh four aggregation subnets, which means uh there are um uh I mean four aggregators aggregating uh

one I think it was uh 64 or no 128 signatures per slot. Uh, and then these these four proofs uh that the the proof of each aggregator generates needs to be recursively aggregated into a single one. Uh, and as a reminder, we are using 4-second slots. Uh the results of the devnet was basically that the time to finality was around 11 slots and eventually the CPU contention made this uh the the thing the the finality delay increase over time. Uh but this was a

good start for this since we know at least this is like working somehow. Uh and now to more interesting stuff. Um The next experiment we run was a 1,000 1,024 validator devnet. Uh for this one, we actually run on multiple servers, one node per server, one server per node and split in three data centers from Hetzner. Um we are using different configurations for aggregator nodes and non-aggregating nodes.

The aggregator nodes are using double the number of cores as non-aggregators at 16 and also double the RAM with 32 GB. >> [sighs] >> And the devnet configuration used for this one was again 32 nodes with a 32 validators per node for a total of 124. We have this time double the number of subnets, eight aggregation subnets for a total of 128 signatures aggregated each slot per per aggregator.

We again are using 4-second slots. And the the result we got here was more interesting. Basically, the time to finality stayed somewhere around 24 slots. Uh and we also saw a lot of reorgs at 11% of blocks being reorged. Uh we did a an analysis of this and we found basically that the time to finality seemed to increase by some amount. As you saw, before we had 11 slots of finality 11 slots to finality with four subnets. This time with eight subnets, we got double that or nearly double that with 24 slots.

This what This means that basically recursive aggregation was slow. And aggregators took a long time to aggregate finality votes. This was the symptom of time an increased time to finality. And basically votes are being included here on aggregator. Uh In this diagram, basically the x-axis is the slots that passed after a block was proposed. And the y-axis is the weight um trying to justify that that block.

We can see We can see here that there are like multiple small steps. These are unaggregated votes that have the weight of only a single of a single subnet. Well, and over time you can see that there are some bigger jumps that are for um when multiple when unaggregated vote is actually included with multiple with the weight of multiple subnets. Uh these There are three stairs because they are three random blocks we chose for this analysis.

Um yeah, after we go above the dash line, it's the block is finally justified. This has to be repeated again with a with another block for for this one to be finalized. So, the time to finality is double the the the time you see here to for justification. And this explains the This explains the time to increased time to finality. Now for for the reorgs, we did an analysis of the gossip arrival times. We can see here that signatures and aggregated payloads are mainly arriving on time, but uh blocks are

uh all of nearly all of them are arriving uh after they should have arrived, um which is uh the first interval of the slot. Here um each interval is a 800 milliseconds, and we can see that uh as blocks are packed with more and more attestations, here the the time to arrival of the block gets higher and higher. So, as the network degrades, um there are more attestations going around, and more are being um included in blocks, and this uh basically leads to this degradation that ends up giving us a surprising 11%

reorg blocks. Um Yeah, just to drive the point, the we did an analysis of the number of uh of the percentage of blocks that were reorged um depending on the time it took to build the block. Um some of the blocks that took over 5 seconds to build, most of them and 80% were reorged uh while the ones that arrive on time, none of them got reorged. So, this is consistent. Uh and now to basically the reason that uh the blocks arrive late is that block building um took too much time, and this

is basically due to the block proof generation. The the block proof is um generated by the by packaging the blocks with with a the different aggregated payloads that were sent by the aggregators into a uh with Limium, and this um the difference between block production and signature aggregation is mainly that the aggregation signature aggregation is done by aggregators which use a more beefy servers the non-aggregating the the validator servers are less have less resources so this explains why a bit of that difference the other part

was when we made this experiment block production wasn't as optimized as this same message recursive aggregation that aggregators do. Anyway we try this again with eight seconds slots this time doubling the total slot time. >> [sighs] >> And this time with the limbiam version that use black. With the same configuration hardware configuration same definite configuration this time only doubling the the slot times.

And with this go we got some better results doubling the with with better limbiam performance and also doubling the time we have to build slots we got the time to finality down to I think it was seven second seven slots. This is like 14 slots with four second This is equivalent to 14 slots in the previous four second four seconds per slot configuration but it's still a better time to finality. We also started focusing on the safe target delay you can see here in the in the right of the of the day of the metrics.

The the time it the the distance from the head to the safe target is a kind of measure of the of fork choice health. If the safe target distance is too is too big then it means the fork choice votes aren't being included on chain, and this also delays the time to finality. Here we managed to lower this down with also some client optimizations.

Um and then for analysis of the gossip arrival delay, uh we can see here that two things on the right, on the aggregator payloads, we can see there are like some some groups arriving um on time and others arriving late. This is mainly due to the second group being the recursive aggregations, and we do a bit later in the slot. Here it's not differentiated, but still we can see that aggregator payloads the main ones are arriving mainly on time.

Then signatures are also arriving on time in the middle, and on the left the other interesting part is the the arrival time of of blocks. We can see there are like some some um groups that are arriving on time, and then some other groups that don't arrive on time, and it repeats like a pattern. Uh this is again what we saw that since we are using different um since [clears throat] block proposal generation is expensive, and we are using different hardware hardware for uh validators and aggregators, we see um that basically when the we

for choosing the proposal in each slot, we are using the round robin, and aggregators are allocated like um one next to the other. So, these periods where block proposal blocks arrive on time are when an aggregator got to propose. Since they have a better hardware, they managed to propose on time. When validators have their chance to propose, they take too long, and their block arrives late. Uh to try to mitigate this or improve this, we uh tried out some changes to the aggregation pipeline.

Again, we are using uh we are targeting 1,000 validators. Uh and basically the change we made to the this This is a breaking change to the to um Well, not a breaking change to the consensus, but uh it is for the for the gossip layer. We were We were We tried moving the block proof generation to the aggregators to um better to take um to use their better hardware for this.

Uh and this basically means that aggregators continue continue continue sharing uh their aggregated payloads in the slot, but afterwards, when it's actually time to build the block proof, they they are the ones that build it and share it with with their peers, so that the proposer then can choose one of these block proofs and just package it into a block and provide it. We are using the same hardware configuration as before, uh with 32 servers, and the same network configuration as before.

Um and with this configuration, we basically reached the three slots of finality we targeted. Uh we can see here again the safe target delay uh measuring fork choice health is lower than one, uh which is a an artifact from the metrics, but sha- this is basically um a one slot distance, which is the the um stable state of this. Uh and then for analy- analysis of the aggregation of the gossip arrival times, we can see basically that um >> [snorts]

>> uh signatures and block signatures and aggregates are arriving million time. Here the the the time each of the this message issues has to arrive is 1.6 seconds. So, and as a second as a second in plus this this change in in architecture gave us, we have we can see that the resource usage of a non-validity non-aggregator nodes was drastically reduced. Can see before that we we were even using swap to try to build to build block proofs. Now, since

since validators don't have to build the block proofs, they they have a lot of resource to spare. But, I and in aggregator nodes, the resources the resource usage increased, but not that much in the case of memory at least. Um and the findings we we got with this was that well, we reached the time to finality target of three slots. With eight second slots, this is means a time to finality of 24 seconds.

We also saw we also saw no reorgs in in this in this configuration. And the we also got the the the re- reuse reduced uh non-aggregator resource usage. And now for the final experiment, this one we did some days ago. We tried bumping the total validator count to 4,000. And this was using the same servers as before with the same configuration the devnet configuration except for the fact we bumped the

the number of to 4,000. And we also did some changes in the way that I think the more most important one was just bumping the lim beam. Uh this one still using Blake. But it's a newer version. I think we measured it to be like two times as fast. We also do some some aeration parameters. Make delaying a bit the the block proof generation.

Later into the slot. Uh to give more time for aeration. And we also made um more most like a patch for the networking part. Um making a raters a coordinate a bit without using network. Just a deterministic algorithm to skip um propagation of redundant proofs, which was um delaying this a lot. And again, we we reached three slots of finality three slots time to finality with this configuration four times as many validators as before.

And uh for gossip arrival delays, we can see this is still uh within bounds. Um as a recap, we managed to stay at three slots to finality 24 seconds with no reorgs. Uh but uh four times as many aggregators. Uh aggregation still uh takes too long. Uh this was because we we managed to reach three slots of finality uh tweaking a bit the aggregation parameters.

Um we uh aggregation needs a a a longer interval in the in the in the link spec configuration, I think. Um to give aggregators more time to to share these aggregates. And also, we started finding problem with the network propagation of proofs. The network started getting um saturated. This with the many proofs that aggregators managed to pump out each slot.

Uh this is with eight subnets. If we bump this if we increase this even more, we'll quickly reach a um uh it won't be possible, I think. Uh so we need to work on the on the network part. Uh and finally, uh some next steps we we think we we want to take. Um the first one is we want to start working on the couple consensus. We look forward to it. Uh for um splitting the fork choice and finality votes should help with this um safety distance as I said before.

Um we uh one thing we have in in beacon chain sensors is that since fork choice and finality votes are travel along and need to be aggregated, they um the time to finality starts depending on I mean, the the fork choice the the time for a a block to be have enough weight to stay as part of the chain uh gets a couple with the with a finality duration, which is slow. Uh we are also looking forward to any more linear VM optimizations.

Um And we also have to try we also have to try different parameters for the on this architecture. Mainly so mainly changing the intervals of the the the size the the duration of each interval inside the slot, but also trying to scale with more validators, more subnets, and lower slot times. >> [sighs] >> And as I said before, we also have have to look for networking improvements to tackle the propagation of aggregation pills.

And that's it. Any questions? >> [applause] >> Thank you. >> Um in regards to the networking side of things um do you have any more insight in regards to what was the main bottleneck there? Was it the case that because you were running so many validators on a single instance, it was the hardware that was um saturated? Or like what was the what was the root cause? Do you have any more insight?

>> Um we didn't have that much trouble with signal to propagation, which would be like the the main symptom for having like too many validators in a single machine. It was mainly the the aggregated payloads that saturated network. Um we saw this as just aggregates arriving late most of the time. Um like with the example of uh from four subnets to eight subnets, we actually saw um the that a lot of aggregates started arriving late.

Uh while before with four subnets, uh I mean the they were running in a in a single machine, but um we didn't see anything of the sort. >> Okay. Yeah, I'm just a little bit surprised because it's a single machine or like a cluster on um you know, some cloud service uh which you'd assume has reasonably good um like networking, far better than like a home operator. >> Yeah, um we actually tried the four subnet configuration, I think with the multiple servers, and we didn't see anything of the sort. Yeah.

Yeah, the problem started when we jumped to eight servers. >> I have one question. >> Thank you. Um where you think the most um juice can be taken out of this like the optimizations? Do you have optimization in mind? >> Uh well, I think the next step is just the networking optimizations. One thing that was mentioned before was that maybe we could start to uh try looking at this um providing some metadata of the like the the big maps of the aggregations of the aggregations before

the proofs. So that um uh we can avoid this redundant propagation we saw in the in the one of the last experiments. Uh yeah. >> Okay. >> I have one question for number of validators. So you tried a 4,092, right? It's cap of the rings back, right? >> Yeah, uh Yeah, 4,000 is actually a cap. Uh we tried with 8,000 and yeah, we found quickly that cap. I didn't have that in mind.

Um but yeah, 4,000 seems like the limit right now. >> But uh in practice, we have a 1 million validators in current is I am so does it works? >> With 1 million? Yeah, I don't think so. >> Oh, yeah. I forgot the correct number, but yeah, a total validator has joined the current mainnet. So if we increase the number of validator for this uh how can I say Uh does this work for how can I say? How?

So the bottom neck is the EVM and the proof generation things. And if we get increased number of validator, the the uh uh relation will more and more increase. So I think it's not work for bunch of by data sets. Don't you think so? >> Um I think I don't understand the question. Sorry. >> I I I I tell you later. Yeah.

>> Okay. Um yeah, just to add um the main ways to scale the number of validators we can run in a devnet is trying to scale like uh the number of subnets with 500 validators per subnet we can reach uh reasonable aggregation initial aggregation times. But once we you start increasing the number of subnets, you start having more rounds and even more rounds of recursive aggregation and that slows things down.

And also each of these rounds uh adds more uh propagation at the networking level. So uh that starts becoming a problem. Right now I think that recursive aggregation it's like at at this point it's fast enough. The problem the bottleneck starts being network propagation and here we are running like one aggregator per um per subnet. So once we deploy this in mainnet for example the the main idea is that um anyone can become a an aggregator.

Uh I think right now that's like the idea. Um in when that happens you'll start having a redundant aggregators in the subnet and so this increases uh the the amount of bandwidth you need and the number of uh like duplicate payloads you you duplicate aggregations you'll receive from the network and we have to cut that down to uh to scale this. >> Any other questions?

Okay. Thank you. >> [applause]