Scaling Ethereum's Networking Stack: Mempool, Blobs & Congestion Control

Transcript

Introduction: “We Scaled Ethereum… Not Really”

I hope this is working. Yeah. So, I don't know if you have noticed but last year we were scaling Ethereum and we have achieved quite a lot and now we have scaled Ethereum. So now we can change to agentic AI right? Uh not really uh scaling is not working like that. So uh we are scaling and scaling continues and large part of the scaling is uh scaling actually the networking stack behind Ethereum. So I will be speaking about this. I'm not going into too much technical detail. It will be a little bit technical. Uh I hope you like that. So uh I have a slide about the uh Ethium

Ethereum Networking Stack Overview (EL vs CL)

networking stack. Uh in a nutshell what you should know is that there is the execution layer there the execution uh uh layer clients uh they form the peerto-peer network and that is the consensus layer the consensus layer clients and they form the network. Each one have their own peer-to-peer functions. Discovery, managing peers, syncing the uh uh the network, uh syncing the the chain, syncing the states, syncing the blocks. Uh meool on the E side, getting the transactions uh and then on the consensus side discovery, P management syncing, uh diffusing blocks, doing attestations, uh doing the blob dispersal and then sampling these bops. Now, uh this is part of the scaling already.

Each one of these uses different protocols. I mean not each but we have a bunch of protocols and I won't be going into the details of these protocols. Uh just notice that there are many of them. We have discovery protocols V4 v5. We have that P2P on the execution layer side. We have P2P on the consensus layer side. Uh and there's also connection between the two clients running on the same node. Uh this is the engine API and then all this is running in an environment right. So we have minimum nodes requirements. nodes are different.

Nodes, Geography & Environment Constraints

Uh we have a bunch of nodes, 10,000 nodes and they are geographically dispersed. There are nodes in Africa, there are nodes in Australia and in many other places. And then this is the system that we want to scale. Uh and well the problem with scaling is that when you are scaling any of these can can become a bottleneck. Uh so it's it's not like yeah we scaled one part and now we are done. uh but because then another thing pops ups as a pops up as a bottleneck and uh and then you have to uh think about that. So there are lots of things that we have to uh think about there's this menu of of things that that we have to scale and uh well I will be speaking about these so let's start with three of them discovery pe management

and syncing the network discovery. So discovery you see discovery when you are starting a node uh it has to attach to the network and uh the problem with discovery is that discovery is slow in the in the current implementation. Uh at least this is what I heard. So uh we have solutions uh to make it faster. Uh first of all we have a new protocol. I mean it's not new. It's all new. Uh that was V4. There's V5 and V5 is is much better in this extent. But not uh the CR clients were not all on on uh V5. Now we are moving them to V5.

Discovery Protocols (v4 → v5)

Uh then V5 has a feature which is called topic search. Uh which is very useful in some cases. Uh for example uh when you are running a node on mainet uh that's fine. You are finding peers when you are running a devet or the test net or a small network with the as a technology but not mainet. Uh then you have a problem. Uh you cannot find nodes easily. topic search is helping in that and then there's a third thing uh that uh actually this is not slow so I had uh this post about showing that uh you can actually do the discover the whole network of 10,000 nodes in uh 50 seconds um like you get to almost 10,000 in 20 seconds. So so discovery is fast and discovery you need that when you start

but this discovery also gives robustness to the network. So whenever there is need for a change you need that to to change the network. Next one is peer management. Uh we appear to network. Uh the CI nodes has peers. Uh the uh E not has peers and in EL we had this problem. There's the same problem also in the CL. Uh that it was hard to find peers who were saying yeah yeah come and let's talk to each other because everyone was full. Uh and we have a few solutions here and we are implementing them one by one. So inbound round pe separation that was already there and then we are adding new ones.

Topic Search & Faster Network Bootstrapping

Uh what you see there on the curves is how we are speeding up uh this process uh in in uh last year actually. So it took this is not in seconds this is uh more in I mean it's the time scale is not 20 seconds here but here we are in minutes and you see to ramp up a node it took 30 minutes before and now it's uh basically in 5 minutes it's damping up to full capacity and then there is sync. So to sync uh the thing to know about sync is that you're starting a node and you have to sync it and if that's so then you cannot start a node basically and that's not good for the network uh because then they start to die out and

Peer Management Improvements

then they would die out. So you have to have uh a sync uh which is fast enough as we are scaling sync also has to be faster. So we have to work on sync. Uh and there are two use cases here. One is when you are just starting up a new node and then everyone is helping you start up and that's the easy case and then there is the hard case when there is some problem in the network and then everyone has to sync or let's say not everyone but there are few nodes and then 9,000 nodes hanging on them and saying help me help me uh this is where we need help and uh this is where we have to change and uh this is where the solution is doing u uh Yeah. So syncing similar to uh to be turned having uh

Faster Node Ramp-Up (30 min → 5 min)

doing things like uh small swinking. So we have already parts of this uh uh in the stack. We have some uh some signals there. Uh it's not yet implemented. Okay. So after we looted these three things, uh let's look at the main part uh which is the me pool. So the meool uh scaling the meool. So we already scaled uh a lot Ethereum you see it um and we want to scale things like blobs per slot and we want to scale also execution we want to scale uh the gas per second.

Syncing Challenges & BitTorrent-Style Ideas

Uh unfortunately if we want to have uh more ops per slot we need to send them to meool send these transactions to meool uh and we have to send more if we double the capacity then we have to send twice as much and the same happens with gas per sec. So when we are scaling execution uh it's nice that we are now at uh 60 uh but it also means we need more transactions. These transactions have to allow to the memp pool then have to be distributed uh and then have to be put into blocks. Uh so everything has to scale. The two figures that you see is is kind of what's happening in meool like the proportions. On the right side uh the lead is the blobs uh because there I'm showing it as weighted by size

Mempool as the Next Bottleneck

and on the left side the big one is not the blobs uh because there I'm showing it in the in the in the numbers. uh but we are scaling both numbers and uh and and blob the numbers of blobs so while the proportions might might remain the circles will be bigger uh so I will be speaking about a little bit about what's happening in the meool and and then uh about the kind of proposals on the table to to scale it uh this year so what's happening in the meool me is is getting the transactions and then distributing it between all the nodes uh ideally uh the question is is it distributing a transaction to all the nodes uh and and

Blobs, Gas Scaling & Traffic Growth

the answer is yes. So if I today look at the meool and uh check if take a given transaction then basically everyone has it. This graph is showing uh on the down on the on the x- axis uh yeah uh what's how many nodes have a given transaction and basically what you see that everything is on the right side for every transaction that I'm looking at everyone has it basically and that's cool that means that the meool is basically fully synchronized so whoever is building a block they have the same picture uh they can build the problem is that this is not not sustainable so if when we are scaling there are more

Fully Synchronized Mempool Today

transactions and then everyone would have to have more and more and more at one point we cannot sustain that. Uh another thing to to interesting thing to look at how do the me works is how much time uh transactions spend in a meool. So you submit your you create your transaction you submit into mempool uh then it's living in a mempool for some time then it's getting into a block then it will get finalized and kind of this is the time between uh when you submitted generated the transaction and when it it's uh finalized. So this is the first part of that the journey and what you see is that first part of the journey is actually quite fast. So what you see here is kind of individual

transactions and a curve is how they were getting diffused in the in the mool. So from zero nodes to 10,000 nodes basically. So when you see it shoot up it means that in 0 something seconds uh you send sent it to into one node and then like 90% of nodes had it and then it has this curve uh saying yeah and then the rest is is getting it uh slower. This is a different view of the same thing. So it's the same curves. I've just pushed them to the to the left side. So this is uh how fast the transactions are are getting spread if you if you put the it uh the starting point uh to the same. So when uh to the

Transaction Diffusion Speed

to when you have submitted the transaction. So why this was if I was able to go back was I yeah uh here the zero point was when the transaction was getting into the block. So the guy most to the left was submitted 17 seconds before and it was an unlucky one uh who needed 17 seconds to get into a block. Here you see the unlucky guy was submitted and then going there. Uh anyway the message is like diffusion is fast in the mur. is working and again you can look at it this way and uh here you see uh now not individual transactions because there are many of those but the statistics so

Blob Propagation Performance

here what you see different types of transactions uh and the important part is including blobs which are huge are being diffused in the mool very fast uh blobs are a bit slower than the other ones but still uh crazy fast I mean compared to the slot time which is 12 So you send it by the time there will be someone building it's being diffused and that's good. That's what we want except that we want to scale. So we want to push more bops. We want to have smaller sort times uh and so on and so forth and then these girls will be much worse. So we need to do stuff.

So what can we do? So as I said today uh the meool is is working very nicely. we have this fully synchronized meool among the nodes at least if your time granularity that you are looking at it is is the slot granularity and that's what we do actually every slot someone is looking at the meool and saying hey I can build stuff um and this is true for both for the normal transactions and for blobs tomorrow this cannot be true to tomorrow uh we cannot keep this notion of fully synchronized uh meool simply because we want to scale the transaction amount.

Why Full Synchronization Won’t Scale

It's more traffic for every node and nodes will not not like it. Uh we want to keep the nodes requirements stable. So by definition nodes should have a partial view of the meool and this is first something we we need to solve for blobs uh because blobs are big and then uh when these agents start to send a little transactions uh we will have to solve it for the transactions as well uh because there will be many of them. Okay. So when you want to scale the map, there are two main issues. Uh we typically think of the first one uh which is we need to reduce the bandwidths. So uh we need to send the same amount of traffic but we should somehow use less bandwidth

Reducing Bandwidth: Peerdas & GetBlobs

uh than what we send and we solved it in the in the consensus layer. Uh we started with podas. So P is reducing its only sampling. It's doing a probabilistic sampling for the blobs and it's reducing the bandwidth requirements approximate approximately to to 18 of of the actual uh data amounts that we are sending. And then there is another technique which is called get blobs. Get blobs means since the E already distributed these blobs, the CL is not even distributing it. is just looking at the EL execution layer client sitting next to it and saying do you have it? Yeah, I already have it. Then it gives it to him. So the CL is not even dealing about this with the distribution and with this

the CL bandwidth consumption is even lower because the job is done by the execution layer. Uh where the man poolool the execution layer instead have all the blobs uh and it actually have has the blobs which make it to the chain and it also has the bops which doesn't make it to the chain. Let's not forget the that in the meool you have more traffic because some transactions are not making it to the chain right and you also have to handle those. So the meool is is kind of disadvantaged. uh so we have to solve that last that lower part we reduce it in the CR we have to reduce the traffic uh in the in the execution layer in the M poolool

Execution Layer Bandwidth Pressure

and then uh the second problem which is uh less known is that uh well don't look too much at the figure that's kind of a old uh networking telecommunication technology figure uh but the the thing is there is something which is uh which you can call the congestion collapse So you put more in more more and more demand more and more traffic into network and at one point it cannot handle it and it collapses uh until you're not scaling you you don't have this issue right because you set your performance point to a point where where you are safe when you're scaling uh you're continuously trying to hit that boundary and the problem is if your

Congestion Collapse Explained

system is not the right one then when you are hitting the boundary you have this catastrophic collapse. Um you can have this at the system level or the system doesn't function anymore but you can also have at node level uh your node cannot keep up maybe it's at the defined boundaries of of uh uh of hardware requirements but maybe you are running something else on it or maybe it's overheating a little bit or whatever and your node is getting this collapse behavior. So well we have proposals to solve both of these uh for reducing the bandwidths we have the spas block pool proposal uh which is already in EIP which is reducing the overall bandwidth in the

man pool. So it's kind of a combination of two techniques uh it's one of one of them is is kind of is probabilistically fetching only some of them. So only part of the network one six of the network uh will have the full blobs and then you have also the other kind of technique which is it's using only uh it's taking only the other notes are taking only part of the uh of the blobs so only the cells that they need actually only the cells that the consensus layer client will ask them later on because that's why they need it. uh this is an EIP uh let's see how that evolves then there are other proposals here so one is different proposals on on sharded bop pool uh you can do sharding in different

Sparse Blob Pool (EIP Proposal)

ways so we are speaking of blobs horizontal meaning the blob direction you can do horizontal sharding in different ways uh you have to decide basically based on what you decide which the blob belongs to uh you can do it based on the source sess of the sender you can do it based on the hash of the transaction or you can come up with other things. Uh these are being debated. Uh vertical sharding which is kind of doing the cutting the blob in pieces and and uh shards by the cell index and then this is still blobs and then you can do shard in meool. So once you realize that you have to also scale the meool uh you have to sh me as well. Uh it's not something we have an EIP yet uh but that's something we have to think

about. And then there is the other problem uh the congestion collapse. So again there are some uh ideas here uh EIP877 uh which is basically saying uh when I'm announcing a transaction and for this you have to know how it's working in the in the meool in the meool I'm just announcing I have this and this and this and hash and then you as a node you're seeing he has those hashes he has those hashes he has those hashes and then you have to decide what you pull but you only see a hash so what you do and the problem is when there is congestion you just see more and more hashes popping up uh and you don't know what to uh you can just fetch randomly basically. So this is adding more metadata so that you can intelligently fetch fetch uh which has the advantage

Horizontal vs Vertical Blob Sharding

of uh uh you will not have nons gaps. So uh transactions which are behind gaps which cannot be included and you can also filter by address. So uh you can do things like I'm only focusing on part of the address space I'm instance in this address etc. Then there's a second one uh which is specific kind of congestion collapse. Uh we have this mechanism which is called uh replace by fee. So you sent and this is specific for blobs. You send a blob transaction and then uh you have to replace it. You have to substitute it and it's typically because you have to give different fee caps. Why you have to do that? Because the market fee is not what you were thinking it will be. uh so

Smarter Transaction Fetching (EIP-877)

you have to replace it and when you have to do it in the current uh system you have to resend the whole blob transaction and then it's getting redistributed again and that's very wasteful uh so this EIP is solving this problem it's saying when you are changing all the fees let's distribute only the new fees and not the whole block content uh to the nodes and then there's a third proposal uh which is called B tickets uh which is well uh you require tickets to submit transactions. I'm not going to detail of that. Uh that's being actively worked on.

Okay. Yeah. So this is how we scale uh DL and then I have two little slides for you uh uh on scaling other things. So well CL directed uh one is is what I would call the low latency P2P toolbox. Uh so I was putting this together uh last year. Uh it's not that I invented any of these. Uh these are techniques known in pebb research for some time. Uh the good thing is we are starting to apply them and kind of make them fit the system that we have.

Replace-by-Fee Optimization for Blobs

So there are a bunch of techniques uh segmentation, scheduling and priorization using quick uh better transport than TCP doing encoding lead on fountain quotes network coding even uh or optimizing the overlay uh which means currently you can easily send messages back and forth over the ocean. That's just feels like a waste. Uh but optimizing that is is actually very difficult. uh because if you optimize it too much then you have these clusters of networks. So uh anyway there are techniques on that and uh and we can start applying all these one by one.

Low-Latency P2P Toolbox (QUIC, Encoding, Overlay)

Some of them we started segmentation is already done partly with the cells and cell level messages. Uh encoding we use the encoding uh quick transport is is being introduced in all the CL clients. So yeah all these are being uh done and last uh we have pas which is doing its uh its columns of uh and it gives us uh this eight times uh gain in the uh distribution of B content. Uh we have research on on other techniques. uh I was doing this this fas version two which is kind of a combination of a bunch of techniques um targeting uh a bunch of blobs 256 blobs so it has block

Peerdas & Future Networking Improvements

level encoding it's kind of making make it's making it such that when finally we decided that this will be the content block content of a block then the system is is helping that happen uh and it has many low latency techniques uh Yeah, that's it. Plus many other things that others are working on. So the networking part is is about to improve and I hope we can provide more scaling to Ethereum also this year. Thank you.