devp2p, libp2p, ethp2p - Untangling Ethereum’s Networking Stack

Transcript

[music] [music] >> Yeah, so apologies for the last minute change like this presentation was meant to be given by my fellow colleague Dennis. Um but the main idea is that we are going to try to walk through through the networking stack of Ethereum like uh pretty much like the existing um networking layers and also like talking a little bit about the future like what's kind of coming up. So again um small introduction, the execution layer uh networking stack, we will talk about the consensus layer networking stack. Uh we will introduce like what used to be like the roadmap for what will come in next which is F P to P and I mean as a

team that has been uh pretty much working on networking layers for quite a few years like we will also like to do some kind of opinion or where we think it was going to be the best solution for the future. [snorts] So um yeah right now like um as you probably already know like there are two separate uh layers that you have the execution, you have the consensus. Each of them is running on a completely different networking stack. Uh what do I mean with this? Like despite they share the same needs from the application perspective uh both networks need to discover other nodes in the network, they need to use a transport layer to be able to communicate and understand each other.

Um there are given set of protocols they need to use also to understand like what the other peer is actually requesting to you. Um and the idea is that the application layer is also using these layers to uh yeah coming or yeah communicate with each other. Um what I want to realize or what you I would like to transmit with this slide is that um they have completely separate things although uh you can see that there is Discovery five and Discovery four as a kind of middle term uh but the idea is that despite they need the same things they are not using the same things uh and this is a little bit like where we are going to about in in talk. So, having separate networking stack in the end, what creates is that separate

overlays. Basically, Discovery 5 in the execution is running its own stack. It's discovered in different peers and it's completely detached from what's going on on Discovery 5. So, there is a big overlap between nodes in both stacks, but the overlay it's not shared. Um, yeah, this also goes again to Dev P2P versus Li P2P. Um, Dev P2P is using their own way of communicating. It's creating its own peers and peers don't talk to each other. Like the only connection between the execution and the consensus, uh, at least from the node operator perspective, is that they communicate through an API in between. So, both happen differently.

So, let's start a little bit, uh, from the beginning. Uh, Dev P2P was like the initial solution for having like a peer-to-peer networking layer stack, um, back then in 2015. Um, the main idea is that again like you need to discover nodes. This is or used to be done by Discovery 4. There's currently a roadmap to shift from Discovery 4 to Discovery 5, which is, as you can see by the numbers, is like the second iteration. Um, and the idea is that it's academic a DHT style, uh, network like discovery mechanism running over UDP. Like once you are able to discover nodes and they, I mean, this DHT is kind of keeping some metadata about the nodes that let others understand like how they can reach you out and connect with you. And once they

can reach you out, you can use like this, um, LPX communication, which is basically like a handshake coding so that you can transmit information. Um, there's always like this initial hello packet where a node tells the other one which set of protocols you are using. This is pretty much saying like, uh, okay, if I want to transmit a block from you or to you, um, we pretty much need to agree on what's the actual structure of the block. And this is what we understand of protocols sort of like our sub protocols like, uh, there is a protocol to exchange blocks. There is also, uh, yes, I don't know, a set of requests so in case you want to back sync, uh, the missing slot that you have.

Um, so this is pretty much what the P2P was uh doing. However, like, um, since you already know, Ethereum has changed a lot since 2015 and especially like the merge was like a significant milestone because most of the critical things that were kind of shared between the beacon chain and the execution chain like kind of shift everything to the consensus layer, like the this beacon chain. Um so this pretty much left the execution layer uh only well only um in charge of the execution payloads. Uh what does this mean? That um the execution layer or like the execution client is mostly taking care of of transmitting transactions, keeping the state of the execution layer so that they can actually process and validate

like what's what's actually could be included or not. And this can be also used for basically like sync any existing um uh blocks that were in the past so that you can recompose the state and also like this is a highly critical thing because the consensus layer still needs to validate the payload of the blocks that are getting propagated. So as you can see like the there is a really big interconnection between layers, but the networking stack is completely separate. Um like following the main IT rule is that if something's working, you should never change it. Um and again like because most of the complexity has shifted to the consensus layer, this means that yeah, there is almost no maintenance to do at this layer unless there are vulnerabilities and things to

to cope up. But um I don't know like I think there was already a talk from Shaba taking that there are still improvements that could be done at least at the structure level. Like for example, now um they realized that it's actually more useful if we extend the transactions metadata while we are propagating them um as a way of making yeah, being able to make smarter decisions. So the second iteration was like uh back then when they were working on the consensus layer, uh they realized that okay, we built a completely custom networking stack for Ethereum back then and libp2p came as a completely different or radical solution which was more like um more general and modular solution where different clients, different networks, different

whoever will be able to share the stack. With the idea is that if we unify the forces, eventually this is going to make the maintainers simpler. So Ethereum was one of the embracers of this libp2p networking stack uh and it was adopted by the beacon chain back then. Like the the result that we have today, like I don't want to say that this is only happening at the consensus layer. There [snorts] are different clients, different language implementations also in devp2p, but at least in libp2p this is not only shared by clients on Ethereum, but they are also shared among other networks and yeah, it's it's pretty much trying to build something together that all kind of maintain.

Um so yeah, this this slide's pretty much highlighting that there are quite a few teams and quite a few implementations of libp2p. Um as I said before, like uh libp2p has been kind of taking more and more and more critical um data that has to go through it. Uh there are basically two ways where data is flowing using libp2p. The first one is GossipSub, which is the main broadcasting protocol. Uh you probably heard about it, but it's basically the way that um a node can have like the structure meshes with some few nodes and that's the way that uh Ethereum blocks, attestations, aggregations, columns are distributed. Like pretty much have like a a connection on on each of these uh topics. And then

broadcasting is not the only thing that needs to happen at the Ethereum level. Like eventually you need to know who are you connected to, so there is like always some metadata to exchange and this is generally happened through request-response uh RPC calls. Uh and this is also like in case you realize that uh your there is a node that is uh kind of sooner or like it has like more updated versions of the state of the chain, uh you can pretty much like sync from it like using this request-response protocols. So [snorts] as a critical component, like I want to pay like special focus on GossipSub, which is the protocol driving most of the critical things in in this consensus layer. So

it used to be a generic protocol basically to propagate information around. As I said, like there are messages, messages could fly by. The issue is that again Ethereum has been evolving so much since 2020 specifically to the point that at the beginning it was something that was working. However, we've been putting more and more pressure on the protocol by 2024 blobs came. Suddenly we had to send more messages which were high payloads. Let's say like the average beacon block it's roughly 150 KB. Blobs can be up to 128 KB and we are having quite a few of them. So, this was just putting more and more pressure and

the gossip sub protocol pretty much kept the same. So, at some point we realized that we needed to evolve as the application layer was demanding more the protocol should evolve with that. That's why we introduced I don't wants pretty much saying other peers that we already have something so that we are not wasting bandwidth. However, like this requirements from the protocol have been increasing over and over like to the point that the protocol and the application has evolved together like because the protocol was kind of enforcing things from the application eventually this led to yeah both evolved together.

This was pretty clear with the peer data solution like we wanted to scale blobs but we knew that gossip sub wasn't able to just deliver the payloads. So, ideally what we did was like splitting columns or erasure coded columns and then we were kind of able to split the load among different meshes. Like I hope this is clear otherwise you can catch me up later. And once we had this schema of yeah propagating smaller chunks or at least distribute the workload we could actually increase the blobs. So, I really hope this is making like a clear idea of how both the development of both things were happening together.

There are still drafts like I think that this might be like this partial messages upgrade was pretty much proposed like one year ago like I think it was 20 like yeah summer 2025 Uh the idea is that at some point they realized that um when you are getting pieces like for example the blobs. Blobs of something that has is is going into different channels. Like you have uh blobs going through the network through the execution layer like um someone can post a transaction and which is uh which has like a blob attached to it and this is already fetched by some of the nodes and when that transaction gets included into a block they realized that some of those blobs were actually present there and this is just a way of reconciling on a smarter way instead of having to broadcast the whole payload. I know that

I have just some pieces of it. Um again the idea is that this has been the evolution but we have come up with this future where we are realizing that again like gossip sub is kind of keeping us from actually scale a little bit more. So I want to talk a little bit more about the future like what's actually open and how or what's the roadmap uh to to be able to serve all these demands from from the application layer. [snorts] And this leads us to what was up to date the main component put there like uh in the roadmap which was called if p to p. Um it was so you saw that we went from something completely custom for Ethereum even though it's also used by other networks uh to something more general purpose which is shared among other

networks and the idea of f p to p even though I'm still not super sure of what actually meant. Like it's an idea basically but the idea would be that we will go for something custom again. Like uh we've reached a point that having a general purpose protocol doesn't give us enough. It's just like a black box that it's harder to control. So f p to p again the idea of f p to p at least uh how we got it is it was okay let's try to make a protocol or a networking stack that it's actually efficient doing what Ethereum needs.

Um so again like the biggest challenge in Ethereum is that we are constantly demanding more from the broadcasting layer. In generally bandwidth is the biggest bottleneck because if we had unlimited bandwidth we could just literally just waste it as we wanted. But, unfortunately, especially if we want to keep uh homestakers, this is not available. Um so the idea is that what I want to highlight here is that this can only get worse in terms of bandwidth. Like we want to have like larger blocks with a higher uh gas limits. We are constantly trying to increase the blob limits. Um even if if the protocol is already kind of stressed right now, we want to even make

it worse by reducing the slot time so that we can have like faster finalization. And eventually, what we want is also to increase the number of like information that we do we want to transmit around which for example like uh execution uh proofs. So again, like the main idea that FP2P wanted to raise up is that eventually like what we realized while as like I I spent a lot of time like debugging and uh pretty much taking a look at gossipsub and it was pretty clear that the bigger the payload we have, the harder it is to actually broadcast it around. So FP2P kind of identified this and what is proposing is like why don't

we do a protocol which is pretty good at erasure coded broadcast by itself. So the idea is that you split a big message into chunks and you are able to uh have like erasure coded version so that you don't necessarily need to have the you don't necessarily need to receive all the pieces, but you just need half of this extended payload um in order to just receive the information. Like this is pretty much like increasing not only the propagation speed and uh the resilience of the network. Um something that we well we like uh Raul back then identified is that gossipsub is a little bit naive in the sense that you identify a message with an ID, but

there is no link between the content that you are sending and the ID rather than a hash. So in this case like the what I want to raise like our say here with the slot awareness is that we can play smarter with the metadata that we share about the message. Let say we know for example that uh talking about attestations, we know that attestations have a the a finite number of validators that need to commit to it um [snorts] on every slot. So if we were able to have like a a slot aware metadata, basically we could help the application layer reconcile which attestations you are missing without having to spam each other with attestation that we might already have.

Like this is also a way that we could maybe reduce the number of duplicates that we receive which eventually just saves bandwidth. Um again, I'm not extremely sure of what F P to P meant but I think like the roadmap was to just ship something built on Libp2p and try to just make Libp2p like lighter over time. Um again, this is my understanding so I might be wrong. Um but one of the things that uh F P to P was kind of pitching is that because Gossipsub or Libp2p was such a general purpose used by so many clients, eventually like the governance of okay, I need a new feature like how do I land it in Go-libp2p or Rust-libp2p. So this process was getting

heavier and heavier because different teams were kind of needed to agree on on what was actually needed and how to implement it. And I mean last but not least like it was a shiny new name. Like again like Gossipsub didn't have many upgrades for the last four years and eventually someone was actually pushing to have something new and uh we thought that we shouldn't underestimate like on a place where we knew that something was a problem and someone was proposing a solution that was also like a big uh driver to keep the conversation going on. So just before I run too late on time, um again like Problab has been I mean it used to be part of Protocol Labs like more so our teammates like were actually part of the development uh of Gossipsub.

So we do think like that we have enough experience on the field to maybe give an opinion of what are the takes that we will take when we want to I don't know shape maybe the future networking stack of Ethereum. So the idea is that just shipping like as I told before like this um partial messages feature built on top of Gossipsub even though it's something okay it's not a simple solution but it's just a new feature. It took a year since it was originally planned until it landed and I'm not sure right now how many clients have implemented it but yeah it's something that takes time and it's a critical point where we cannot underestimate the effort

that it takes to actually write the specs implement it test it and actually roll out a working version. So if we want to shape a new networking protocol stack we should take into account that this is not going to happen like super quickly and that this is a multi-language effort where interoperability is super critical because as I said all the critical components of Ethereum in terms of at least messages that need to fly around are going through these protocols. So our proposal would be that you can't change everything as a plug and play no let's just ditch libp2p and let's just build something from scratch like the more components in a client

that you want to change straight away the harder it is going to be to implement it and to test it out. So the idea is that we would like to pitch like a solution where you just reuse what it's working similar to what devp2p is why we kept it right now um while still improving that missing part which is like we can't keep with a black box as Gossipsub is for some most of the application layer. Um I put here like for example the governance influence like it was raised back then as an issue like having like things were taking too long to land into the actual protocol but at least in the last year and a half like Ethereum has been the main driver for libp2p at least in Go and Rust which are the the bigger

used uh uh uh uh Um so yeah so I mean, I think that uh Daniel would probably talk about Wagle, but there is already like I I wasn't sure like what was the road map until few days ago, but the idea is that there is already a lighter version a lighter message sender version for broadcast, which is actually built on libp2p, which was actually the plan that we will have proposed to go. So I'm kind of happy to hear that Ethereum already went into that direction. [snorts] And yeah, this is pretty much the talk. Like I just want to end up with this analogy. Like I know that some people might not really like um SpaceX and all the fancy things, but it's like this is pretty much like an analogy.

Like you can only identify what's wrong when you have it done. So I think about libp2p and all these broadcasting protocols as a big chunks of modules that as soon as you go evolving, you cannot go from the first version to the last version in one go. Like that's a big lift. That's a big refactor and eventually you will realize that there are things that you went wrong. So my idea will be to pitch that we go gradually changing those modules in a safe that it not it doesn't necessarily need only to help Ethereum, but probably also like the whole web three um ecosystem.

So yeah, that's all. I think I'm on time. Yeah, happy to take any question if there are any. And yeah. >> Round of applause for Michael. Thank you so much. We have a lot of questions. Let me start with the first one. Can Ethereum improve network bandwidth without introducing a third networking stack? >> Absolutely. Like so okay, like I spent quite a lot of time like digging into gossipsub and there are a bunch of improvements that we thought were really easy to implement. So it's I don't want to ditch gossipsub, but it's a rather naive protocol which there are many improvements that you can easily do without having to ditch it.

Again, it's reached a point that maybe at some point it's easier to just do a refactor, but uh there there is there is the possibility to improve the bandwidth without having to replace every single step in in the protocol. >> Can we use protobuf as a message format instead of JSON messages which reduces the payload size since PQ payloads are much bigger? >> So protobufs are already been used. Like there is no JSON structures going around at least at the networking layer. So there is this simple serialization happening and then like all the content is already compressed.

So yeah, that's yeah. It's going to be happening anyways. >> What do high throughput alt L1s use for P2P? >> That's a good question. So so if we think for example at Solana or Monad that the idea is that Ethereum still wants to preserve at least try to don't deanonymize validators from the protocol perspective. So Solana has a really unique way of identifying like validators need to say which IP address do they have. So basically you have okay, there are this number of validators, there are this number of IPs

and the way they achieve that high throughput is basically because they know who they have to send the messages straight away. Like in the case of Ethereum this is I mean this can be a decision made if that's the way we want to go but if like unless you actually make all validators say which IP they are on, there is no way that this is going to happen. Like and I will suggest to don't go that way now. I think that we can improve the throughput of Ethereum without having to give away like some at least privacy of the of the validators. >> Plus ZK EVM and PQ proof sizes and signatures will be much larger. What's the ideal P2P stack for this?

>> I mean what's the ideal stack? Like we don't know. Like that's the point. Like we don't know if it's simpler. Well maybe you Daniel know more about this. The reality is that messages are just going to larger and we will just have more messages. So ideally we still need to shape like what's the ideal stack. Like I cannot answer that question right now. But it looks like being able to erase your code it and send it over at least when payloads are larger that's that's the way to go. >> Okay, why not gRPC and just JSON RPC?

Isn't gRPC faster? >> So I think this this again like there is no JSON formatting happening at this at the working layer. I guess this applies more to the EL CL API communication. So yeah. >> Okay, next let me see. How much headroom do we have left in the bandwidth of a home validator?

At the current rate of ETH improvements, how long until the networking layer is a hard bottleneck? >> So just to be like we saw like there was a presentation by Migalabs and that's like analyzing like the blob usage in the network. So like the ETH Panda up has been doing a really great job like understanding like how much could we improve like increase the blob count. And since the last update like like we increased the blob target to be 14 and what we realize is that we are barely going belong like above this 14 mark. So we are still not yet or we are not still there

yet to the point where like the bandwidth is a bottleneck. Again like just the fact that we are not reaching the limit doesn't mean that we can make it better. Like the fact that I mean we can also don't know what's the average bandwidth on validators. We we know that big part of them are running in data centers and yeah that can keep up with the propagation. But yeah the idea is that we are still fine now, but we should like again, like this protocol developments are something that take too much time to develop, so we cannot wait until the bottleneck is actually there to work on improvements. That that will be actually my my take, but yeah, we are we are still not there. I don't know.

>> Thank [snorts] you so much, Mika. Time is up, but I think you the whole week you have a booth or >> [music]