When the Network Shuts You Out: Measuring Resilience in Permissionless Networks

Transcript

[music] Hi hi everyone a very warm good afternoon. I do wish to share first of all thanks so much for taking your time and coming today to attend my talk. I do wish to share over the last two three days uh my initial intent to was to start with a presentation but I do feel that I would like to share a demo first because that is what helps kind of relate to what the topic is about and try and understand what we are trying to do at scale I just wish to be uh give a quick intro I'm a maintainer at lip and IPFS and our projects are used for emergency coordination across multiple countries there mult I would say at this juncture close to almost like 30,000

th000 projects in total that depend on us on GitHub and uh what I'll be doing over here is I'll be walking you to understand how we work in permissionless networks how we do observability at scale using a very interesting tool that has been developed by one of our core contributors Yash and the name of the tool is Luminar so what we would like to do in the first 10 minutes of the talk is to really understand observability at scale using this tool try and understand the types of attacks that happen when we're talking of permissionless networks. So we'll try and infuse some of these attacks that is civil attack uh eclipse attack and network partition attack. We'll try and observe like how our networks behave when these uh

attacks happen whether our network is a long live network. It's a self-healing network. We'll try and observe that. We'll try and also see how the latency works. So before we get started on understanding the measurement ideas and analytics, let me share a demo video. So here what I'll be doing is I'll be sharing with you a mesh network which is essentially how all our Ethereum nodes also function. How the different uh types of swarm nodes also function, IPFS nodes, you name it like LP2P nodes used by all the consensus clients whether it is lighthouse, prism, teu, uh nimbus, loadar uh and even I would say the lean ethereum clients they all function. So I'll be taking this example. I'll be playing this video. Here you have like

50 nodes uh over here and uh what you will see is uh essentially mesh attacks like seel attack at a specific point of time and you will see how the delivery propagation average hops mesh degree the overall health of the network and essentially the way the network is structured kind of heals itself. Second is the eclipse attack which you are seeing at this juncture over a set of 50 nodes and you will see how the message passing which is essentially gossip sub 1.1 in case of ethereum gossip sub 1.2 and 1.3 in case of partial messages and 1.4 in case of uh organizations like logos and vaku and gossip sub 2.0 in case of the newer architectures that we are seeing in case of ethereum they function. So essentially what we are

trying to imagine is this kind of a network where all the nodes even uh like they kind of understand each other they are able to discover each other we have like peer discovery at a scaled so essentially these are the bootstrap nodes. So bootstrap is one specific method of discovering each other in a system. But then there are different other ways also. We have randu where we are trying to do like communication by namespacing. We have random walk and we also have essentially multiccast DNS which is for local setting. So when you talk about permissionless networks these are this is like a quick 101 like these are the things you should kind of relate to. One is peer discovery, second is peer communication, third is peer routing which is kadima dht. The fourth is like how you do optimization at scale

and then if there is a security attack like how the network heal itself we have to understand that. So this is one demo which I wish to share. There's also one quick video which I had for this specific uh uh like for this specific talk and that essentially is that when you have a peer-to-peer network I mean how do you do observability not just for the nodes but also for the agents. So there is a very clear phenomena heading where we call as like the internet of agent movements and in that kind of a scenario like how does gossip sub function I mean how does peer scoring grafting and all the different scenarios we discuss uh across the three different uh peer communication methods that is gossip sub flood sub and grid topology uh they function when you try and do a

sewell attack you do an eclipse attack and how do they kind of recover from that so these are essentially the first two quick videos or demos which will give you some clarity on to what we are trying to do here. You can see an mechanism where essentially whenever there's a heartbeat. So whenever there's a heartbeat and when you see a node panicking think about it that you don't have to bring any attack the node will not be able to communicate with each other and the network will fall down. So this is not an application layer attack. This is more as an infrastructure layer attack. I think this was one of the questions that was asked to us over the last two days that what's really the difference uh in case of an infrastructure attack and an application layer attack. So Binance attack is definitely not a infrastructure layer

attack. In case of Binance, uh it's more on the application layer. In the scenario that I just explained, this is at the infrastructure layer. So here what you have to do is you don't do have to do anything. You just have to change the arithmetic values of how nodes function. If there is an integer overflow, what will happen is node will panic. If the node will panic, what will happen is the node will get out of the network. And if the node will get out of the network, we are saying nodes will not sync. And you will have like all the D5 protocols like for example a exchanges like Morpho I mean they are the nodes getting out of sync or you'll have consensus clients that is your lighthouse your TU your uh prism uh which is one of I mean these are like some of the popular clients or even I

would say Nimbus loadar grand bean all of them going out of sync and that basically means that the whole distributed network will be in a panic mode so in order to avoid that I would be discussing some of the pointers I'll also be discussing some of the application ations in reference to when there is government surveillance when there is when there are governments trying to do a like try to understand how you're communicating and you're that kind of using observability as a tool not for scaling the network but also to kind of uh cause trouble in certain cases but in some cases it's like the uh bad actors not from the government from like members who want to create panic uh within the network I mean how do you kind of mitigate some of these scenarios is what we'll be discussing over the

talk so I'll just like to very quickly share that if you wish to learn more on this subject uh I I would definitely recommend you to read uh some of the tools. So we have developed a very interesting project universal connectivity to try and understand how this functions at scale. Uh we have this implemented at this juncture in at least 15 different programming languages. So if you are a Goore developer, a rust developer, a Python, JS, you name it, or even a Nim developer which is used by the logos network, uh this is one of the uh tools uh that I would strongly recommend that you contribute and you participate in. Uh I would also love to draw your attention that privacy at the infrastructure layer is something uh it's in progress. So it's not something where I can say that we have a full

understanding of things. So here I would like to draw your attention to some of the things we do at LIP P2P. uh would like you to try and study our spec guide and participate in our weekly meetings. So so these are like some of the things which is like the 101 for this talk. Now I'll get started with the presentation and I'll uh take around like 10 minutes uh for the same. So uh I'll just switch over uh to the slides just one second. Yeah fantastic.

Okay. So I do wish to share the initial discussion. The reason why I always start with the demo is so that you are able to visualize what I'm going to present essentially and that will give you an idea what I'm trying to suggest. So I don't want to make it like a theoretical uh exercise uh this entire session essentially if you really look at what the current scenario is the real world condition is very very different from the test net assumption that we have. So when we write like a research paper or do study on peer reachability which is essentially autoonat or if we talk about nat re traversal which comprises of circuit relay hole punching and autoonat again for measuring the connectivity. So sometimes we look at scenarios that okay there will be no filtering there will be no such hardened version of gossip sub that will be at

play but in reality I do wish to share that there is something called as bootstrap infrastructure isolation which is always there. We have noticed with some of our case studies on the ground for even for emergency coordination and I do wish to share that here many governments have been very supportive but what they still say is that there you will have these cases of infrastructure isolation scenarios and you have to figure out ways to deal with it. So I would love to share that attention also like when we are measuring the resilience of a network I mean how at the same time we can understand some of these pain points and try and work towards it. I'll also be sharing a bit on what do you really mean by dynamic infrastructure filtering and also selective filtering and what do you really mean by IP blocking and how you

can mitigate that by a newer architecture a newer architecture in terms of how you represent the nodes. So we'll be discussing a bit on node representation. So first let me share with you that decentralization does not mean resilience. Decentralization essentially is the architecture. Resilience essentially is the outcome and I would say coordination is really the key over here. By coordination I really mean to say like how the different nodes they talk to each other and how you have set them up in a specific scenario and how they kind of ensure that they are long lived over time. So that is something which we wish to discuss and even if the nodes are not discussed and even if the nodes are like not able to communicate with each other how we can still ensure ways through which some of the nodes can ensure that the network is healthy and hygienic. So

this is some of the things which uh I'll be trying to touch base on. Second, I do wish to share there's a lot of discussion on quantifying censorship resistance. I had some fantastic friends from Nyx Foundation uh who were trying to share some of their work and they have very nicely quantified some part of it. I do wish to share that these are some of the metrics through which you can essentially uh do quantification of censorship resistance but there is something you need to also understand that it goes beyond that and I'll share I'll take an example of one of my friends from Lisbon uh who is working with us in improving the peer communication system. So we have come up with this idea that okay let's try and work out spread. Spread is like a privacy layer built on gossip sub. Now it brings the privacy metrics very

nicely but where the problem comes is sometimes in the performance. So how do you keep a balance between privacy and performance is one of the things we are trying to figure out in terms of not just some of the projects that have been developed using our uh networking stack that is lip p2p but also at the consensus client level. So one of the things we feel is that when you talk about crops uh we kind of relate to the study of Zezy. So Zimz essentially was used for emergency coordination and one of the things we realized over there is that we supported different formats. So we had like Jack Dorsey's bit chat we had noster and we also had lip P2P all supported together and we realized over there that many of times nodes are getting shut every now and then whether

they are in bootstrap mode or whether it's multiccast DNS mode. So in order to ensure that the nodes stay alive we tried and we tried and extend the overall multiiformats representation of the node. So we focus on multi adders. We also looked at multi-ash, multibase and multi-codec. And we tried and combine all these different semantics at the format level and also tried building a data composibility layer which is IPL essentially which is for ensuring that nodes have relationships with each other and uh that means it's a content addressible node and we tried and learn some of these things and see how it works in terms of this specific application that is zam there's another example like that that is called where we followed the same thing and we realized that this is very much similar

to what vitalic and what everyone was discussing the first day and what the whole theme of this conferences that how we can ensure crops at scale. So I would love that some of you you try some of our implementations try and make some simple projects and universal connectivity which I shared in the first demo is something which might be of interest specifically when you're trying to run scenarios uh across different networks. In fact these two apps were essentially built with the concept of universal connectivity. uh I also wish to bring your attention to newer ways of doing delegation and uh I would say uh update recovery at the client level. So there are newer uh I would say proposals that we have been working on. We would definitely encourage you to join our bi-weekly spec meeting where we try and

discuss these news new ways of doing uh delegation representation communication publish and subscribe routing. Uh in routing I do wish to uh thank David Majeris from Stanford University. I mean he has given us an excellent paper since 2002 which is academia DHT. I mean that is one of the standards on which most of the routing systems have been developed. And I do wish to share that some of the things that I'm mentioning over here some of you might find that hey what are we really discussing? We have been reading about uh my dear friends Justin Drake's bunker mode and he talks about quantum attacks but I do wish to share that quantum attacks definitely is of great attention but along with that we also have to consider some of the infrastructure attacks where you don't

have to attack a specific system you just have to ensure that the nodes they don't talk to each other if the nodes don't talk to each other the uh consensus lines don't work and if the consensus lines don't work I mean all your D5 protocols and all the uh nice things we have been working in this ecosystem they don't come to fruition. So would like to share this specific point like if you're trying to quantify the adversarial overhead for Eclipse resistance you also have to look at a hardened version of gossip sub and uh I'm taking in reference 1.1 version which is used by all the ethereum clients right now but I'm more than happy to extend it to 1.3 2.0 and would love to have your thoughts in reference to that. Uh next I do wish to share we have like gossip sub under adversarial

load it should always stay clean. So this is essentially like when you have different cases of attestation flooding control frame frame spam and choked pipes. So these are like some of the scenarios we have seen that gossip sub works in many conditions but then I have some of my friends from EF who have come up with this new topologies that is grid topology and flood sub which also works very nicely in certain cases. So it's not like one shoe fit all sizes. uh when we talk about peer communication it's like you have to look at a specific scenario you have to kind of advise what you need to really use uh in terms of pubsub in terms of communication and routing and try and come up with a strategy which through which you can ensure resilience at scale next I do wish to share there's been a lot of

discussion on privacy uh in uh over the last uh two days and I was uh kind of uh doing a self-reflection on where privacy and observability they converge and where they diverge and uh here I do wish to share one of the spec work that we have been working on which is the spread specification of gossip sum and we have done some measurements over there and I do wish to share with you that for nodes regular nodes it's absolutely fine but for nodes which are representing agent nodes so there is another uh challenge we have to deal with which is like how these different agents they talk to each other across different network boundaries and enterprise boundaries or even I would say user boundaries. So in that specific scenario it will require a different concept of thinking through how we do gossip serve and how we do

flood ser and how we do grit topology. And I would love to open this to further discussion on what your take is on how we can keep a balance uh between the privacy and observability part where you do like a visibility without surveillance and you're not really getting too deep into someone's uh understanding of the nodes. Second, I do wish to share uh this is a very simple schematic of helping understand how active observability has been considered at this juncture when we talk about some of our friends from different organizations on on how they're kind of observing the overall health and hygiene of decentralized network. Again, I do wish to share that keep the first part of my talk uh in your mind which is that decentralization is just the architecture. Resilience is really the

outcome and if you talk about coordination that's how we get it. So the if you keep that in mind and you try and uh look at this semantic you will try and understand that this kind of makes sense basically we are observing we are understanding the we are detecting we are kind of recovering and we are measuring at scale. So I would love to share some of the uh scores that we have done uh quick is basically a transport system. I think many of uh my friends were confused like what are these different uh elements that we are looking at. Web RTC direct when I'm talking about it is the web art direct v2 now. So we have come up with a new spec there was a deprecation of web RTC direct so lip P2P uh friends uh in different implementations we they came up with a new spec of web RTC direct v2

which is essentially when we talking about uh communication and then we also had like TCP and quick we also support websockets and websocket secure and now Bluetoothle since uh bit chat is very much focusing on that and these are like some of the measurements we found as a help profile scorecard and I would love to have your thoughts uh on the same so I do wish to share uh some of the uh some of the key findings. So one is essentially the universal connectivity project which I again I shared in the first slide. I've taken the example of the Python peer concept. Uh we have that implemented across uh I would say 10 different projects. Uh Cisco's outshift project which is internet of agents.

They also interestingly use P2P library uh at their uh directory level stack for the agents. uh this is more for uh userto-user communication uh where we found this kind of a measurement of unified testing. Uh in reference to unified testing do not confuse it with universal connectivity. That's essentially how you do interoperability across different implementations. Think of it like lighthouse and prism talking to each other or like the different nodes talking to each other in real time. uh at the infrastructure level. Uh we also have some of the other things which I do wish to mention that we have extended this whole idea of uh peer-to-peer universal connectivity across different implementations and have interoperability across that at one of my friends working on what what I

call as decentralized GitHub and uh he actually shared some of the interesting ideas on how we can scale it uh with interoperability measures. Uh second I do wish to share that open source definitely is a operational resilience mechanism but uh I do feel that it's more of like an ecosystem play over here which is what I do wish to share. Uh if you have to have long live networks who can self-rebuild themselves you have to definitely consider in that perspective that you need to have code inspectability but also have a capacity where if nodes have fallen off the radar they can come back and also focus on organic traffic not like traffic which is like just there to prove that they are there basically. So we are looking for an active engagement of nodes not like nodes which are like off the radar and not working together. These are like

so our work essentially is very much aligned with the Ethereum's crops priorities and I wish to thank my friends from EF for giving us a very good set of maintainers and very good support on how we scale our efforts across 17 different implementation languages and how we really ensure that some of the ethos and values of Ethereum they stay within the infrastructure layer. I do wish to also uh share one specific thing that in reference to the work that we have done we really believe that networks should be able to rebuild themselves and uh here you don't need agents uh to come into play you can just have nodes talking to each other and rebuilding themselves and uh I just wish to leave with one specific thought which is that have a stack which is modular and which essentially understands the

identity piece the routing piece the multiplexing piece the communication piece the transport piece Also I would say the security and the secure communication channel piece working together together in cohesion. Thank you so much. >> Thank you so much Manu. Sure. >> Now we have some questions. >> Let me start first things first. >> Sure. >> Is there an economic value attached to the general P2P gossip? If not, how it's secured against a funded attacker who can set up many long live nodes with which could get good ranks.

Fantastic question. I actually wish to clap for this question. I do wish to share that we are big believers of sustainability and I do consider this scenario where if you have an attacker mode specifically uh I'll call them I'll also classify certain nation state actors who have the resources to uh beat against odd. So here I do wish to share it's a question of network design and clever design and uh you you're absolutely right when you're asking this question we can be like 99 times correct but if one time we are incorrect I mean uh that basically the whole proposition of peer-to-peer nodes and decentralized infrastructure and the entire schematics that we discuss uh that goes into rupture so I do feel that uh in reference to how you do gossip sub you take it as one specific architecture

it's not just the only architecture there are multiple versions of gossip sub also. So here you can be specific and you can kind of extend and adapt and be interoperable across different gossip sub and if you have an attacker you always have to stay a bit ahead of the curve. So it's always like a like for us I do wish to share with you in the last 3 months the number of security issues in gossip sub that we are working on that's equivalent to the total number of issues that I have received in my career at lip2p. So that is the kind of like scenarios we are dealing with and I do wish to share that yes uh I completely agree with you that we are learning on this and we would love to have your participation for the same. Yeah, >> great. How do you threat network health

and performance in a ZK system where data is private? >> Sure. Sure. That's again a lovely question and I do wish to classify that under the three headings in which we are really focusing our energy at this juncture. One is definitely the whole ZK scenario and second essentially is uh basically the quantum scenario quantum resistance and third is the agentic scenario where you have internet of agents. This is in more interesting because I do feel that the ZK part essentially is about ensuring the like you don't reveal the data. So in case of anonymity what we do essentially is that you have to have some trust. So we add another layer of trust and this is what we have done in one of our spec initiatives. So we have like a newer

layer where you have subscription on the basis of trust and you try and see whether you can do network measurement based on the trust identity. So you essentially are trying to have new identity systems and these are all ITF and W3C standards which you're trying to adopt but you have to add certain layers to ensure measurement at ZK uh ZK front otherwise you can absolutely not just uh like look at doing measurements over here. There's also one more thing I do wish to uh mention and that is that when you talk about ZK measurement you also need to understand that it is one of those areas which operate at a layer which is at the identity front or which is at the autoonet front. What we are trying to essentially do is we look when we do measurement we look at the routing

and the peer communication front. So at on those fronts I do wish to share observability is very much possible. You're not revealing any data. You're just looking at how the data bandwidth propagation and all the other metrics they work in action. So more than happy to further discuss this. This this is a very very important question and it's not like a 5m minute quickie that I have to share over here. More than happy to discuss in detail. Yeah. >> Is there any use case where flood sub will be better to use instead of gossip sub? >> I again a very very interesting question. Thanks so much for asking this. Yes, there are multiple use cases where flat sub would be of interest and uh would have some very interesting implications uh over gossip subub and I

do wish to share there are like multiple scenarios that come to my mind but if you're talking of ethereum project so there I do wish to share that there's a very interesting initiative which we are trying to do uh which is like try and do optimization of popsub so in some of the optimization of popsub there's a very interesting approach by the name of RLNC which is random linear network coding by optimum from and which is uh one of the you can say benchmarks but then there are two other approaches uh which is read solomon strategy and uh then the other one essentially deals with how we uh do pubsub optimization at scale so in many cases when you're trying to do optimization over a local network over a specific scenario so there essentially you are trying to use flood sub over gossip sub but again this is I would

love to understand the specifics of your uh like what kind of uh use cases you are looking at in terms of uh the Ethereum network or outside the Ethereum network and I'll be more than happy to share a network design for the same. Yeah, >> I think we one more. Can the network resilience be measured through automated testing by simulating node failless network latency and connectivity disruptions? >> Sure. This is again I do wish to share that this is one of the reasons why we are developing Luminar and universal connectivity project. So essentially if you're talking about uh node failures so what we are doing is we are injecting uh these attacks in case of luminatu. So the seel attack and network partition

attack they were not happening. What I was doing was I was injecting these attacks in a scenario of 50 nodes 100 nodes and up to 1,000 nodes and yes you are absolutely right that we can do testing over here. We can definitely measure resilience but there are certain scenarios where again you have to do some optimizations. You have to see like in reference to like the latency and the connectivity part. Absolutely you can do those measurements. But when you look at node failures, we have just covered specific scenarios. There are many other scenarios that could be possible where I would love to have your thoughts, your discussion. But for these three scenarios, absolutely we can measure network resilience. There's also one more thing I do wish to share which is again another question which is missing over here. Uh which was sorry which uh due to lack of time we cannot discuss

and like uh which is how we are approaching the whole quantum resistance effort. So on the spec front I do wish to share we are doing the very similar thing very simple thing we're taking an ITF standard which is PKIX and essentially what we are trying to do is we're trying to make it interoperable with all the quantum algorithms that are out there which is MLDDSA and all the other uh algorithms and just ensure that we have interoperability or even crops based understanding at the infrastructure layer. So we don't have to make algorithms specific to like different different blockchains. we'll just make it useful at the infrastructure layer that you can come and plug and play your different algorithms and can have your quantum clients talking to the non-quantum clients. So that's the other question.

It's uh like just listed over there but just wanted to refer. Yeah, >> thank you so much. Unfortunately, time is up but people have more questions. I think you can go and ask um [music]