Deep Dive into libp2p [EN] [Ethereum Protocol Day]

Transcript

[Music] Yeah. So, yep. Um, yeah. Today, deep dive into lip P2P. Um, the question marks there because it's not exactly very deep. Um, so yeah. So, what's LIP P2P? All right. Yeah, this disclaimer, this is my attempted journey um trying to learn more about LIP2B myself. So if you spot any mistakes, go ahead. Okay. So what's le PDP? Um it's actually just a collection of libraries um for P2P network communication. The name's pretty obvious. Uh it's created by Proto

Labs and uh it's used everywhere including in IPFS um the Ethereum consensus client and a lot of other distributed systems um that you might or might not know. Um so yeah that's that's why liquidity it's basically powering all those systems and yeah so in this talk specifically I'll be talking about three components of uh lip P2P um the transport net traversal and pubsub uh transport as in how how the bytes how the data is actually communicated across the internet traversal is a little bit about how to communicate or how to find private nodes that might be behind a

firewall and popsup which is the main mechanism in which uh the P2P clients or nodes distribute and listens to messages from other nodes. Um so yeah so transport um so right now um so yeah it's the underlying protocol for sending and receiving data. So you might know of the transport you might heard of it's usually TCP UDP uh or the ones for web pages which is HTTPS and HTTP. So same thing here um lipb currently supports uh TCP as a transport quick web sockets web transport and web RTC. Each of them um supports different environments. uh TCP

and quick usually just supports things that do not run on the browser and the rest usually um works on or usually it's used in a browser to allow you to essentially have a lip connection between the browser and some other nodes or even between browsers and browsers. So this is to allow lip P2P to run in the browser environment which is quite useful to have especially if you run some web application that you want uh peer-to-peer connection. So I specifically want to talk about quick which is um a newer version of um layer 4 protocol I believe um similar to um

well similar to what we have before but uh essentially what quick wants to do here is um or what quick does is it reduces the round trip um for your um request or for your network request. Um so in a usual TCP so this is actually TCP. So in a usual TCP um communication between the client and the server or different clients uh you will need this amount of uh request response. So there's the TCP handshake which is three step sync sync act which is acknowledgement and then another acknowledgement that you have received the aation from the server and then you can do the TLS handshake which is um the

the protocol in which you encrypt your um connection itself and then finally you can start sending data. Um this is what a usual TCP connection looks like and quick because they use UDP itself so they don't need to go through the general TCP handshake. Um it's usually just three oh I think this one round trick so it's the initial um connect and then sync and then you can start sending data. Um if it's a reconnection or a retry after your initial connection, it can even go to zero round trip just start sending data to the um server. Uh that that is what happened. That's also

um possible if you have already previously connected. So this is quite exciting because um you can imagine reducing a round trip on the network. Um essentially means you have less overhead on the connection itself which means you can send data slightly quicker. Uh which if you send a lot of package or you send a lot of data or if you want need to like create a lot of connections this eventually becomes big enough or effective enough to increase the efficiency of your connection. So yeah and so this is from the if consensus pack um back then quick the quick standard was not really finalized. So this is their

reason for not using the quick uh standard and yeah so it's quite interesting that uh they they deconsider this and they talk about hey you know we are not doing quick because there's still not enough support and yeah but very interestingly um since uh our previous speaker binary actually talked about lean ethereum uh in one of the PRs in lean ethereum they actually did talk about using quick as a replacement for TCP. So maybe quick support in in new clients. Who knows? Yeah, you can check it out. I leave the links here. So you can always Google and find it. So yeah, um before we continue, let's talk a

little bit about how LIP P2P does addressing. So uh you might be familiar with how IP address looks like. So that's actually IP address right there. Uh so like 198.51.100.0. Uh there's also the IPv6 version of it which is in hacks. I don't have an example here. Um but for lip P2P usually um they so for this library specifically they encode their connection between nodes with this type of address. So um this is pretty easy to read kind of. So it this this example particularly um shows you how roughly it looks. the user slash the to separate sections and this is called a multi-address format. So you

can see the IPv4 address of the node the port um TCP port uh P2 and it's a P2P with an identifier like this. So this is used everywhere um to denote how to connect to a specific node. Uh this is the very standard TCP connection. Later you will see another type of address a multi address that actually showcase um connection between a relay uh P2P connection. So yeah it's very expressive. So that's why the P2P uses it and also the pro the in in the PP P2P spec you can see there's a lot of mention of protocol ids which are used to negotiate

um what protocol it's using and uh for example this is the circuit relay protocol 1.0.1 0.1 sorry 0.1 uh and this is like an ID that clients use or nodes use to say hey I'm using this protocol do you support it and if it doesn't support it it has a fallback something like that so it looks very similar to a multi-ress format but it's not exactly a multi-ress uh but yeah just fun fact this is used everywhere um yeah okay um net travel so the second part that uh would like to talk about. So um this is not a thing specific to P2P.

This is a thing that is uh used everywhere in our current network. So net traversal essentially so every one of our devices or or if you run if you have internet at home you have a router uh you will have computers that connects to the router and your router usually act as or usually have a firewall and a net address translator. I forgot the n what the n stands for but it's an address translator which essentially translates your um public IP addresses to your private IP addresses. That's why if you look at your home network, all your devices will usually start from either 198 something. So it's a bunch of addresses that starts with the same

prefix and ends with something else. Um so 198.16 192.19 168.0100 1023 such and so and so on right and but your public address is actually different. your public address usually your entire house will only have one public um IP address and that is what the out the internet sees you as or sees your uh connection as. And so when when you send a request to somewhere uh you usually will um go through when you go through internet usually you either have your firewall exposed with some port or you need some relay to do this connection so

that when you send the request and it's received the receiving end knows how to send the request back to you and when they send it to your IP your net your firewall knows how to relay back to your node. um if they if this doesn't exist what will happen is it will just never reach you um essentially. So yeah, uh it's and and net traversal is what we call for technique that allow us to um send to a private connection that your firewall well does not necessarily expose. Um by default our firewall doesn't expose a lot of ports um for safety reasons. Um and if you are familiar with network primitives or networking um usually to expose some um

port or expose the connection to a specific place to the device you need to do something called port forwarding um yeah and that's if you need to do that for every single connection that's becomes very troublesome. So yeah we have this to allow us to send requests to or return requests from devices and yeah just to showcase how how it looks like. So scenario one um if a random public computer try to send your computer some some information it will get rejected by your firewall by your router because your router don't remember anything being requested to that node or to that server and it will not receive it. It might be malicious so you it's not going to receive it. So to actually receive it um what you have to

do is you need to be the one starting the connection. So if you start the connection first um your firewall will allow any response to that specific um connection itself. So yeah this is example. So when you send a request, you actually add a record um that says you know add this record from source source IP source port is your internal uh LAN uh address and then your your your net or your firewall remembers what's the destination that it points to and your uh net will basically tell at this record and when the response comes back it will check if the record exists or

this uh we call it the net pupil exists And if it exists, it will send it to the correct place um as well. So what I want to talk about or what a lot of u pable connection runs into is scenario two where both of your nodes are actually behind a firewall and you can't access it because remember one of them needs to start the connection before you can actually let the connection go through. So if both of them try to send each other things it'll just reject it. So you have to have some way of actually letting them through by either one of them sending a connection first or so on. So that's why there's something called hole punching. And uh it's called hole

punching because you literally punch a hole across the firewall. Yeah. So um to actually do something or to allow connection or communication between two private uh end you need to have um something. So you have to first have no one send send a request which will be rejected and but when you send it this is assuming both they know both of their addresses. So when you're sending um you will get rejected because there's no such thing for node two in in node 2's firewall but node one will add a pupil to its own firewall and then since we are assuming again node two knows where node one is um so node two will then send a request which will pass through since there was an record now in node one's firewall and

then node one can also start sending because now there's an outgoing from node two. So this is roughly what whole punching is and yeah this is in a modern um web internet uh we have something called turn server sign server just to let you know these are all components that allow you to communicate with someone behind a firewall without you thinking too much about it. Yeah. So how does the P2P whole punch? So it's very similar to this example here but this uh you will notice it requires multiple server and each of them are quite centralized. So LIP2P allows you to do it in a decentralized way. Um it's a three-step process. So they use three

protocol essentially autonet uh circuit relate and direct connection upgrade through your DCUTR and this is all so that you can access um nodes that are maybe running uh in behind a firewall you may they can sync they can talk to each other this is for that because you don't necessarily want your node to be accessible in the public internet so first step [Music] This is essentially when let's say I I'm running a node. I want to uh know if my node is accessible. I want to actually um connect to other nodes. So the first step uh I need to know if my notes is reachable or if if they can call me. So

um note basically my note will will tell all the public notes that I can access and tell them hey can you try to call me and see if you can get the call through. If I'm if I'm not recallable by all of them and I get a response from them that they say I you are not callable. Um then I I know I need to go to the next step. If I'm callable by any one of them, it means that uh I'm accessible to the internet and so there's no problem. I can make connections very easily. They can send me updates very easily. Um so yeah, so this is the first step to detect actually if I have a problem. If I have a problem, we'll go to the next step which is uh circuit relay. Um this starts by me asking a public node,

a node that's reachable by everyone. Um the nearest one uh the nearest one is actually determined via DHTs. I will not talk about DHTs today, but it's essentially a distributed hash table. Um so yeah, I I'll tell the nearest node to me for efficiency sake. Yeah, I want to reserve you as a relay. So I will reserve some portion of their computation power and tell them, hey, can you help me relay any message that that's meant for me? And if the node says okay then you'll get something like this. So recall I talked a little bit about multi-ress and addressing. So if node 2 says ah it's okay to use me as a relay as the inter intermediary. What will happen is now I can advertise to everyone else that cannot reach me that

hey I my address is this instead connect to me via this address instead. And you can see in the address it actually tells everyone who looks at it um it's a relay node by looking at the so it's a P2P node2P2P circuit which refers to this and then slash my my node right so the node one node 2 is usually replaced by an actual P2P identifier so yeah so that's how you get or other nodes can reach you via a node two but this is not a permanent solution because um node two also doesn't want to keep handling your your computation. So usually we'll go to step three which is this UR. So this allows you to connect directly and this essentially does the

whole whole punch. Um so yeah let's say node tree wants to connect to node one directly. Uh it will run a connect request through node two as the relay and then it will measure the time it takes for node one to send a connect to this node tree. Um I'm here I'm showing um the case where node tree is a public node. Of course this is a private node. will be two relays. Uh so this is public root. So if it's a private node, you'll have two relays. One for node one, one for node three. So and I'll measure the time. So the node tree node tree will measure the time uh it takes from uh node 3's request to node one's request and through this determine how long it takes to send requests um to each other.

And they they then do something very smart where uh first it will send a sync to node one to let it know um yeah uh we want to connect and once it sends the sync request it will try to call node one at t / 0. This is the the half the time it takes to send the connect first connect request through the relay and time minus the time it takes to send the connect request from node one to node three. So you use this time and then the the the reason why it does this is because it wants to synchronize the time where this note three dials note one and note one does not node three together.

So they want to make the connection at the same time. Um and if that works the whole punch is complete and they can now do a direct connection. um because at if they do it at the same time essentially what what it creates is there is a hole precisely at both of their nets firewalls and and the connection can be done because there's an outgoing connection that's why it's expected to have a incoming reaction uh connection as well. So yeah, that's that's essentially whole punching. Um, this is quite important to actually get to a lot of nodes that are uh behind firewalls for a lot of standard users that want to host their own nodes. This is pretty

much bread and butter, I would say. So yeah, this all abstracted away from LP2. So most of the time we just use it, but good to know. Okay, finally I'm going to take a breath. Tell me if I'm going too fast by the way. Um so yes um finally so pops up uh what you hear the most when we talk about P2P and uh what you will interact with the most as well. This is the mechanism where you communicate or you you talk to other nodes. Um it's called pubsub because it uses publish subscribe. um where you know note one publish subscribe to a topic and and

whenever someone push a message to topic A um you you'll get the response right you you you you'll get the result essentially anything that's new post to topic A you'll get the result this is pretty easier to implement in a centralized setting where there's a source of truth to query the message from but in a distributed settings get a little bit complicated because um it's hard to know where the source of fruit come from. So yeah, this is uh a little bit of a technical details. This is how the schema looks like. Um each so in in lip each message consists of subscription and publish to to communicate what topic

you want to subscribe to and what message um is being published by you. Uh and and this this is roughly what it looks like. Um this is on the specs page. I'm not going to go too deep about it. Uh but yeah, so it just contains this it's an array of two things subscription and publish. Um yep so li P2P currently supports three type of public subscribe uh mechanism. Um the first one floods up which as per the name just floods the network. Um this is very simple. Uh essentially when you receive a message or when you have a topic that you want to publish to or your message want to receive okay most of the time it's a message you want to

publish sorry um you send it to the to every peer that you can see so you send it to every other node that you know of that subscribes to this topic and they then do the same thing. So you can see this is a very complicated graph. A sends to CDB CDB sends to everything you can see. So uh I've notated here but essentially I'll just talk about B. So B received from A and it it in inside its area it can see A which it received from so it will not send it to it again. D which it doesn't know if it already received the message and E. So in this case it will send to D and E but actually A also sent to D in this case.

So there's a lot of redundancy. You get repeated message a lot of times. But it's simple and it works most of the time. It's just very redundant and takes a lot of bandwidth. So that's why someone introduced random sub which is flat sub but instead of sending to everyone I send to a small subset random subset. This reduces bandwidth but because it doesn't care about network topology or how close um another node is to you in terms of your network uh you might get unlucky when selecting your peers. So you might select someone that's very far away and then when you send when you sync with them this this

basically takes a longer time and so and so forth. Um but yeah, this is a simpler way of reducing bandwidth um for your network. That's just the unlucky issue and you don't consider like who's closer to you, who to sing, who is best to sync with. And that's why we go to the current popsup that's used by Ethereum, which is gossip sub specifically v 1.1. There's v1.2 and 1.3. Um but yeah, we were talking about 1.1. So 1.1 uh essentially makes it so that there's a meta data only network or a global mesh and a local mesh. Your local mesh is where full data is sent. So your when there's a pops up when there's a new topic uh

everyone in your local mesh you will send to something like flat sub. However, there's a bigger network that only shares metadata through what we call control message. Um yeah so there's a few sub protocol as well defined in gossit sub that helps you send the message better choose your peers better as well so I'm not going to go through this but this is essentially going through uh what control messages are um these are sent via the uh metadata network and they are basically transmitted through gossips uh we call it gossips because it just notes talk to nodes and like talk about very little bits of data. The metadata data itself

is very very small. So each of them um here essentially has only like one or two fields and it's very small fields as well. So very quickly, so the flow for example, this is the graph flow which essentially means that you are you want you are adding someone into your top uh your into this topic into your local uh network local mesh uh where you will send full messages off and this essentially um yeah just tells them hey I'm adding you to my network know that you're in my network and yeah I will send full messages to you. Um it's a birectional thing. So if you add them to your network, they very likely will add you to their network as well. Um yeah.

So this is the control message graph, right? This before that they are in a global mesh and now they are in your local mesh. Um so this is the I have flow. So what this is is um normally when there's a new message, let's say node one have a new message, it will send to everyone in its own local mesh. smaller network, smaller node groups of nodes and node 4 additionally uh to sending it to all its local mesh or local nodes will also send or gossip a I have control message to any anyone that's outside some some note further away just tell them yeah I I I have this message and if they are interested in it

they can okay I don't think there's battery here Okay. So if I'm interested in it, so this is an example for let's say someone outside of your local mesh have some piece of information that you heard from the gossip protocol. Um you can tell it to give you by saying that I want uh control message. Very straightforward. I want and then they will send it to you, right? Send the full message to you and and let you sync your data. And in this case, if other nodes do not have that message, uh node 4 will then send it to them again. So yeah, this is essentially control message. Finally, the prune control messages tells some node that they want to be removed from your local

mesh. Uh so that they don't want to get updates from you anymore. This is usually used to reduce the amount of people in your local um mesh. Uh so when you have too much, you might want to reduce that to reduce the bandage you need. So that's essentially it. Thank you very much for listening. Yeah. Any question? questions technical.

>> Um, what do you think about I mean why people don't just use other like >> narrowing stacks directly? Uh I think one of the reasons why is prior to lip P2P there aren't a lot of standardized P2P protocol. Uh there's of course there's always the um what's that called torrenting protocol but they are not so torrenting uh no uh so torrening uh but yeah there's some protocol for T2P as well but they are usually for content synchronization. Let's say you want to

download some piece of data that other peers have. >> Bit. >> Yeah. Bit torrent. Yes. So um that how it works is similar to this but that is mainly for content right and P2P is usually used mainly because it's quite mature at this point and there's a lot of work being put into it. Um so usually we will use it but actually Ethereum execution client came before LIP P2P. So they actually have their own P2P framework called defaf P2P which does similar things just with some difference in how it's addressed and all that. So yeah I I think the main reason is just because sleep P2P is quite mature and people use it. It's very standardized

that's why people use it. If there's some other new protocol out there that is better suited people might use that but for now this P2P. >> Okay. [Music] [Music] topic.

[Music] So, I have I want I have local only mesh or global mesh connect.

[Music] I have I I knows local hash local groups [Music] detect topics. I I have I want like requests [Music] topic topic

option option. data structure. [Music] [Music] Pure scoring scoring.

[Music] local measurement. [Music] Global glob global mesh. Okay. Protocol Global Local Global

Usually Rash S the wind. [Music] [Music] P2P um execution P2P

addressing like identify connect note addressing P2P Fore implementation.

[Music] [Music] Sweet. P2P amount of data transfer.

[Music] [Music] PBS protocol to epidemic [Music] implementation.

I see [Music] Parchosp layer. [Music]

patterns. [Music]