Lean p2p

Transcript

me. [clears throat] Nice. Okay. All right. Let's start. Uh, so I'm Tom. Uh, I work at the F in the robust incentives group. I am a bit of a disclaimer. I'm not a P2P networking guy originally. So I don't um don't expect like a deep expertise in this, but I um I got no sniped into like thinking about this problem uh a few months ago. And I want to share some thoughts about what I've been working on basically um and how that fits into the maybe more general blind B2B uh picture.

Okay. So um if we try to start from first principles um there is like this sort of like very simple question we can ask which is how can messages be propagated efficiently across a network of nodes. Um and so when you ask that question you have so many answers like it's been studied by so many people in so many disciplines and domains that it becomes a bit vague and the literature is actually not that easy to pass. Um and it's also like yeah it's not a simple problem either like even if you like ask that question it's sort of like too abstract to be taken seriously and sort of make progress on

it. So what do you do when you have vague questions? You try to act like add constraints I guess and we have so many constraints like uh as you guys know like Ethereum is like a very sort of like constrained environment and we you know we want um the network to be decentralized. So for example we don't have like a sort of like global view bird eye view on the network at all. So we only have like partial local views of the network. uh we have like very strong bandwidth and hardware requirements. Um so we need to deal with that too. Um people can attack uh the network. Uh so we need some sort of like robustness against against this as well. Um yeah

the network is also not stable at all. It's changing all the time and what you optimize for today might actually change tomorrow. Um then we also want privacy. I don't think we are doing too great now. Uh my take is like we should get towards more privacy instead of like I mean some advocating towards like less privacy and more optimizations or efficiency. I I think like privacy is still very important. Um and we have yeah different types of messages being propagated all at the same time. um all like we have validators and nodes doing so many different things at the same time uh that it's not always uh uh easy to deal

with. I think I'm fine. Yeah, thanks. Um uh so yeah, so when we have these conraint constraints, I guess you can just like do our favorite things uh thing to do in in Ethereum is like come up with trma. Uh so here's one. uh it's not an official proven trail. So uh it's just like one that makes sense to me. Um I think you do have some sort of trade trade of space. Uh and you know you can get like a very sort of like super efficient very simple maybe also like like way of propagating messages. Um if you have like a super stable network

centralized infrastructure it's easy. It's not very resilient though because I don't know if you take down the server or the entity then it breaks. Um, and yeah, you can sort of like easily sort of easily get like two of those dimensions to work pretty well, but like once you have the the three together, it gets harder. Uh, so one dimension is leanness or or like simplicity. Um, and today I don't think we are like doing really good on any of these things unfortunately. Like I I don't want to be like too harsh either like but yeah like the the resilience is is kind of good.

We'll see like um at least it's sort of robust in the way we propagate messages which is nice but you know the privacy for example question is is still like not perfect. Um it's not I don't I would argue it's not very efficient. Um, and it's it's not that complex. It's it's not like super not lean. Like for example, the way we propagate messages is like just send it to a random peers. But it's not like simple when you sort of like put it all together. Then it becomes extremely complex and and it's hard to like keep track of all.

>> Um, just a quick observation on this. This bears a certain resemblance to the traditional tradesman's triangle. Fast, good, cheap. So maybe that offers a bit of evidence that this is a trilama. Okay. All right. Well, I'll take it. [clears throat and laughter] Good try, Le then. Um, thanks. And then, okay. So, when you look at [clears throat] it from sort of like an outside observer perspective, uh, you want to know where to start. Um, we because we do have like different propagation mechanisms, right? you have this sort of ego push that is like the most expensive sort of like um yeah you need to propagate one message across like all nodes and then you have like this sort of like fallback of lazy

gossip of like um using the I don't want and I want messages you have two layers um so you I mean yeah you need to deal to like two P2P networking stack in parallel which is not ideal and then of course yeah again message heter heterogeneity Um and then yeah difference between global topics subnets and everything. So yeah it's I think it's for me it was even hard to like engage with like you you come in and you see all of this and you're like okay how are messages propagated through the network like that's that feels a bit crazy. So I I will focus on the one I was not sniped on which is the eager push. Um, I think

that's I focused on this because it's interesting, but also because um there is potentially like a lot of efficiency gain uh to be to be had there. So that's sort of like mostly what I focus on, but we'll see how it ties to everything else because like everything is sort of like interconnected, right? Like you can do like a push to like all different kinds of messages that can be across the EL or the CL. So yeah. Um, and then you can ask like do you even need like a lazy gossip fallback if you have like a really good ego push like so yeah you you can actually like sort of like interate everything.

Um, so yeah um very lean. I think you've seen this meme before right? Uh the idea is can we find like a really good rule that I will want to call soon um because I like it. Uh, and then I'm trying to like fit words to match it, but we'll see how how [clears throat] well that goes. Um, but yeah, the idea is to like can we actually get one rule um maybe with like a we'll see like with the knob to trade off like efficiency versus latency that we can use like across the P2P spike for this sort of like first eager push mechanism. So that's what I'm going to talk about.

Um, yeah. So that's I I'll go quickly because it's not super um well actually some of it is actually very important but yeah we model the network as like a weighted graph um and so yeah nodes are nodes uh which is nice and then the roundtrip time so RTS are on edges so you know the distance between two nodes will correspond to like the roundtrip time between these two nodes. Um and then for the topology we actually treat the graph as a fixed random uh graph. Uh and that was to try to mimic the peer scoring sort of setup we have which is you know we down score peers if they do

something bad but we don't really like sort of like connect to peers because they are faster or anything. So it I was trying to like sort of mimic this kind of random randomness of like having you know a set of like maybe 50 PS you are connected to but like without like an actual sort of like [clears throat] big huristic around why you're connected to them other than they are acting uh they are behaving okay. Um and then local decisions like you don't again you don't have this sort of like bird eye view so you need to like deal with what we have.

Um so each node will only know about RTS from its connected peers which are 50 and you don't have like this sort of like global optimization problem which is actually also difficult to solve but yeah uh okay so what do we try to optimize for in the setting um bandwidth and it's just like average duplicates of the same message received by a given node. Um then coverage like how many% of nodes in the network are actually informed so get the message in the first place. Um and then the latency so that's just like the time to reach I don't know like 90% of nodes. Um and that's the sort of like metric we we use across our simulations.

Uh okay. Yeah. So again what we have today um we have a mesh of like 50 score PS again like using this sort of like P scoring rule I I just talked about the your pushes to just eight random PL um and that's what we are going to focus on and we have this lazy fallback that I already talked about and so the pros is like yeah the random push is like pretty robust like actually really robust I would say because it is random so you can like really trick notes with like fake information which is very nice. The the coverage is also good because you know if you like send to like eight random peers all the time like you will reach like a very sort of like high

coverage eventually and the latency is not too bad. It's not it's not great like you can do better for sure but it's um it's not terrible and like you end up like reaching a high percentage of nodes in not not a super long time. I think the big big bad thing is like bandwidth inefficiency. Like the fact that you just like forward to random peers all the time means you receive a message so many times like the exact same message which is obviously not great. Um and again like the leanness is I don't know I put it in the comments but it's not that that complicated. It's just one. Yeah, just one thing.

Technically speaking, the ear push is not really to eight random peers, but rather peers that have been scored highly, which is a factor of several observations we make, including um the time to first delivery from every peer. >> Yeah. >> So it proxies in latency in a way. >> Yeah. Yeah. >> So it's more informed than just randomly like >> Yeah. Yeah. Yeah. So, eight random PL amongst the 50 scored PL uh and and in the P scoring rule you have some elements of latency. What what you're saying?

>> So, so would you mind? >> Uh so really >> no >> we good. Yeah. Uh yeah. So in reality so what we so Gossip Hub has multiple topics. Uh there is a mesh for every topic. Yeah. The degree of that mesh is eight. >> Yeah. >> Um the peers that we score are those peers that >> support gossip sub and share topics with us. So uh which is basically everybody that is part of um of the network and that depends on the configuration of the client. So every client will have um um a optimal value that they shoot for for

peering with X peers. Right. >> It's around 50. No. Or >> no. So 50 50 would be the target of the EL. >> Ah okay. Okay. >> Yeah. And for example um the prism >> has a default configuration of 70 peers. Okay. >> As max peers. >> Okay. >> Um and usually depending on the type of the node operators will increase that value to something higher usually. Uh so in reality you're peering with usually more peers that support gossip stuff but you're only selecting for each mesh and for each topic that you that you're subscribing to uh you're only selecting eight peers >> right

>> those peers are uh selected based on their scores. >> Yes. Yes. Yes. Okay. Okay. And there is a component to latency in the score. Exactly. But it's quite >> it's it's a secondary. Yeah. It's kind of like a second degree. >> Right. Okay. Okay. Okay. Um yeah, that makes sense. That's that's a good that's a a good call. I think I saw at one point that there was some latency into it and I was like, okay, um that seems to be like not driving like the main sort of like driver for for connecting to. >> Yeah. So just one thing there, it's it wouldn't be uh link latency as observed. It's actually time to deliver the first copy of the message. So every time that receive a unique valid message that we expect that we validate, okay,

>> we and is the first the first copy that we received, we bump up the the score of that peer. >> Okay. >> So in reality, it's kind of like your dissemination time within them with the centrality within within. >> Yes. Okay. Okay. >> But with a with a caveat that the producer >> Yeah. >> is always moving around. So you don't have a good centrality. Yeah. >> Like um absolute centrality measure. >> Yeah. It's rotating centrality. Yeah. Okay. That's >> a question. How how cleish are these peer groups?

>> Uh that's a that's a great question. Uh so there is there is churn and they tend to be somewhat stable. We're trying to get a good measurement of what is the churn of the graph in general. So I don't actually have a good answer right now. But there are like there is rotation. >> So So let me clarify my question. And I'm trying to get a sense of um given two peers, what percentage of peer A's peers are peer B's peers?

>> Right. Right. Um we don't have a good number on that. We do have raw data like as of one month ago uh that we are analyzing currently. So hopefully there's a team probab they're doing they're using this raw data to actually apply a model and to understand how the graph is turning how the graph is dynamically changing over time. So I think we'll have some numbers ideally in this week or maybe next week and yeah that yeah I think that's going to be very useful to make decisions.

[clears throat] >> Nice. Thanks. Yeah, I think the the I mean one of the thing that is listed at the end of my my talk is like the interaction with the peer scoring rule because I do feel like it's super important but it's complex to like even like sort of like think about right now because like you have like so many little elements that go into the peer scoring and so basically I think there are more like two questions like you want to optimize the PR scoring and like PR selection in the first place like and go full in or do you want to keep it somewhat random? Um yeah those are like a big questions >> and perhaps to add to that complexity is that gossip that scoring is applied if I remember correctly to all topics equally.

>> Yeah. Yeah. >> Which every like every object in Ethereum actually has a very different propagation >> path and and characteristics >> and also the the parameterization of gossip sub the parameters are actually shared between all topics. Right. >> And that is um that could change per implementation. >> Uh and that's that's totally fine because it doesn't break the the the standard sorry the the spec. >> Yeah. >> Uh but no implementation as far as I know is able to tweak >> Yeah. >> per topic parameter. >> Right. Right. Okay. That makes sense.

>> Cool. Okay. Uh great. So yeah, for simplicity I did model it like this. So simulation of like you're connected with like 50 random beers and you just forward to eight and you have like sort of a plateau. you can't forward to more than eight, but we'll see that depending on the rule you use, you might, you know, in some cases sort of have like this adaptive threshold that you use to not forward some sometimes. Um, okay. So, yeah, we ran some simulations um to get an intuition of like what we could even think of as like better rules. Um, and so yeah, that's what we use. we had like 1500 nodes in the

simulation. Uh we use the mean degree of 50 again. Um and then yeah for the RT distribution it yeah it actually matters but like yeah um right now we just use like a log normal sort of like distribution um that is like you know [clears throat] sort of like nicely shaped um and we go with that but I I will come back to this because because it's important and then for the metrics yeah I mean I already said it but like we do measure just like the average duplicates um the coverage and the latency be set to 90 just as like a sort of like guiding principle of like yeah um you need to hit more than 90% of nodes to like reach the coverage floor

and latency was measured uh as the time to reach 90% of not um okay so uh I think one thing that's extremely obvious to everyone but like um you have RT awareness that plays a big factor All right. Um those are like two strategies or rules that don't use RTS at all. So the one we have today which is just like um yeah simply simp simplified forward to just like delta neighbors uniformly at random uh um up to eight and then the coin toss will be like yeah just propagate to each neighbor independently with probability

p and you can just like I don't know sweep across that p probability to see what you get. Um so those I mean you you'll see are not like super efficient and then we thought of like two main RT aware rules. Um so the one is very simple like delta fastest just like propagate to the fastest neighbors you can um to the eight neighbors that are like the fast fastest basically with the sort of like smallest RT out. And downhill is uh I'll stop like just a bit on this one because um the rule is forward to nodes that have an RT out that is like um [clears throat] lower

than uh the message you just received. So if you receive a message u and it takes like 100 milliseconds to receive it, you'll only forward to nodes uh that uh uh you are connected to that have like a like a smaller RT that you have a smaller RT with. So this is sort of like that's why I called it downhill. Um the other like Tumba uses WF but it's outdated but basically that came from having this idea of um you know this very known problem of optimal transport where you try to move mass across like nodes in a graph and one of the sort of

like optimal way to do this is just to like have this sort of like gradient descent rule. Um and and that's what uh that's where the inspiration came from. But um so yeah, how is it doing in the simulation results? Um so the first thing is that uh I don't know uh so for the two you know RT unaware uh rules you get like good coverage like very quite fast. Um, so you have like the, you know, coverage on the y- axis and and then the duplicates on on the x- axis. Here you also could get like really good coverage. We can't really see it, but

like with the delta fastest, it's a it actually covers like a lot of ground, but it's really struggling with the downhill rule like here. It's like because it's yeah it's like you you try to forward like going downhill but like given the properties of the graph we are dealing with like you hit like road blocks so you can't like actually propagate that message further and you just like get stuck in like low local minima. Um so that was interesting to see and then for the latency so downhill is not even plotted because it didn't reach 90% of nodes so it doesn't like hit the sort of like floor. Um, but what you can see is it's interesting. It's like you do get a very big difference in latency when you compare delta fastest.

So like just send to like the fastest peels compared to like random or coin does sort of like rules. Um, and so that's one takeaway. It's like okay, it does look like we, you know, do pretty well on coverage for all of these except downhill, but I'll come back to this. Um but when it comes to latency uh you know we can do much less uh like much better latency uh but also with like lower sort of like duplicates if you just have this extremely dumb rule of like propagate to delta fastest PS which is sort of nice.

Um, one of the problem with the data fastest I'll come back to is like the only knob you really have is to how many peers you send it to. Like you know if you want to like lower the number of duplicates, you just have to like send to less peers. Um, so it's not a great way to like deal with the sort of like trade-off you want to make between um duplicates and latency. Um, so we try to go for better rules I guess. And one thing that's super important is the realization that um there's this really cool feature of having a sort of like random set of peers. First, it's robust, right? It's hard to trick, but it's also like at one point you hit like

a per collection threshold. Um and so if you send to more than n randomly, you will hit coverage. um because like because it it like propagates and like the message can actually reach all nodes nodes at some point which is very nice. So the idea was like okay can we use like this random backbone um and add on top for the like remaining PL um like a very optimal sort of like efficient rules. So that's what's what it's showing basically. And so you look at Delta Fastest, you look at downhill and you see how downhill was doing very very bad on the coverage side

um without any random backbone but at some point it just like reaches like almost perfect coverage just because you had like you add some like random peers you actually forward it to and uh for data fastest I mean it doesn't really need the backbone to get coverage um but like it always just like consumes the maximum bandwidth you can ever think of. Uh so here you you can really sort of like see a whole area between like downhill that is extremely bandwidth efficient but that sometimes that doesn't really hit coverage um and isn't doing as good for latency as delta fastest. Um but so basically you have this all area here and here

that you can you want to be able to like sweep across that area, right? And you want to be able to optimize between do I want my messages to be propagated very very fast or do I want them to be like using as low bandwidth as possible. And so what you can do is do a hybrid rule. And so that's what I'm like trying to advocate for because I I think it makes sense. The the first step is basically to set a random backbone so you are sure you hit your percolation threshold. So you can see in many sim simulations that it's always like two or three like after that like you actually hit like decent coverage in like decent license then

latency just like we have today with like the sort of like um gossip sub forwarding to eight P and then for the rest of the P you can just like try and optimize for what you're looking for and we thought of two ways of doing this. I think those are like nice conceptually to think about. The one first one is like absolute threshold. So you only you know forward to a node uh if the RT out is under a certain arbitrary sort of absolute threshold and so that mimics a bit it's a bit different but like still conceptually a bit like reducing the number of peers you send it to like

doing delta fast test but just like reducing the number of peers you want to choose. Um and it's a global threshold like sort of like applied to the the entirety of the network and then you have soon which is okay you have set that random backbone but then you can you know do the RT out um inferior to RT in uh but you can have like a boost of of that to allow more or less pls to be including based on what you want to optimize for. And so what you want to do there and so you can see that it's much more local because it's always based on like the message you receive in the first place whereas like absolute threshold is more like of a global sort

of like okay no I I don't want to like um send it to too many peers so I'll just like shut it down and so you have like a very sort of like you have very different rules uh and we looked at what uh that gives in simulations the idea is always to yeah choose the threshold or the slack S is for stack um but could be anything. It's just like a parameter. Um and [clears throat] yeah, the idea is to choose like how to set those things based on the sort of like preference along the latency bandwidth spectrum. Um yeah, so that was again like the sort of like setup we used and in this setup like we actually see that um uh soon is

not doing much better than absolutely like the performances are like you know you soon like doesn't really reach like perfect perfect coverage like it it does pretty okay but it it never like truly gets to 100%. Um, and then for the average duplicates, like yeah, it's it's better when you have like more slack in your t obviously because um uh yeah, it's it's better when you have more slack in your in your in your when you have more slack. But in terms of latency, you can see that absent still does quite uh better than than soon. So, and it's hard to pass. So you can do

this and here like you basically match the bandwidth you are looking for between the two rules and you see that soon like always sits on top of u of absolute t. So that means it's it's doing a bit worse basically than just having this global threshold. Um and so that's interesting because um we actually noticed that if you change the proper like the RT distribution of the network and you now have like a biodal distribution which you know could be the case in like real world networks because you can have like sort of like clusters based on geographical distributions or even ASN or anything. um then it's

completely different and then you have like much better performance for something like soon than uh something like uh the global sort of like absolute threshold just because like you know for example you never really reach all peers um in this sort of network because like some peers will always be excluded from your maximum threshold. um you always have to consume like the maximum sort of like um latency like you always are like at 300 because you always send to three peers with the backbone randomly. Um and you are doing like a bit better in in the bandwidth but like overall like

the trade-off is is really much much worse uh for absolutely compared to soon. Um yeah and you can see it here like yeah basically you having an absolute threshold in like sort of like a more heterogeneous network like get you stuck uh in sort of like regions you can't like really get out of while soon because it's adaptive and and local can actually like sort of like do quite well even if you have like a a different sort of like network structure and also like RT distribution basically.

Okay. Um, so you see that those are like literally like super early exploration. Like I was running this simulations last night. So don't take it for like a super like extremely wellestablished sort of finding. But it it did feel like as soon as you change and you depart from a very sort of homogeneous network um then [clears throat] absolutely like just like struggles with it and having these sort of like global thresholds on the network. it's just like not really good. And when you just I don't know you go from random networks to like small world networks or you change a bit the RT distribution based on on geography or whatever and soon like always sort of like performs better in this context

just because of its local adaptability. Um and yeah so the idea I want to like try to shield but let's see if if it works is like basically you can use the soon rule and use slack s um across like the latency and bandwidth sort of like spectrum. So if you want very fast message propagation because it's like in the critical path and those are like small messages for example then you can like I don't know just like have a lot a lot of slack and it will converge to having this sort of like delta fastest rule that is just like always super fast and optimal for latency um for for low latency. If you have like very big

messages and you need to like avoid like messages getting getting sent like super many times to the same nodes um then you can like tune Slack towards a more like downhill thing and you'll always hit coverage because you have like the random back backbone but at least you'll be very efficient in how you spend uh your your bandwidth. Uh and so yeah, ideally you could like use that same role and tune this sort of like knob to uh to like yeah to be suited for different types of messages across the network.

Uh so yeah um I think what we really really need as you've seen is better better observability because if you know the propagation rule actually depends on what network we have then we need to know like approximately what network we are dealing with so we can actually make like you know informed choices about the rule we want to use. Um, so I think yeah, some empirical [clears throat] data collection I think is going on already, but I'm very curious to just like get these RT distributions and get a more like granular idea of the topology of the network so we can actually inform these choices. Um, then yeah, I'm planning to

just like run more realistic simulations. I think those were all like Python just like simulations and I think we can do way better with like using running like lip P2P binaries uh in shadow um and yeah I mean yeah at the same time we'll continue exploring I think the next step will be devets and then like try to ship it I don't think like this should wait fall in Ethereum at all I I think like if we can ship it and we have like high conviction it's like a good good thing to do, we should do it soon.

Um, and yeah, that's what I was saying like before I think we had that discussion, but like how it actually interacts with like the ping rule and the lazy gossip mech mechanism is actually interesting to look at and a sort of like second thread that Mario that joined DF recently and has [clears throat] been working on this with me by the way quite a bit. I don't think I mentioned him at the beginning. uh is can we actually prove formally sort of like near optimality for a given rule. So then we can just like move on and use that rule and just like be done with it and and not like spend like like iterate again and again on like finding some some like different mechan like mechanisms for

propagating [clears throat] information across the network. Uh yeah, thank you. >> You guys have questions? >> I do, but I've spoken in a bit, so I want to see any thoughts before I >> I guess not. So I'll take it. Um so a couple of things I think this is uh this is so latency sensitive um gossip is very interesting as as I said we kind of like incorporated it right now is incorporate RT but not really RTT it's actually um like speed of dissemination

or >> uh yeah how how fast we get a particular message um in our scoring metrics so in a way it's already there the I think I think one important factor background parameter in all of this is the graph the site like the topology of the graph that you that like the the simulations have been run with because in reality peers like what is the strategy for a specific peer to connect to other peers in the network. what are you generally like overarching in an overarching way optimizing for >> because in local like heruristics like this

>> you will be sensitive to like what the overall strategy has been right >> um that's one thing second is I do think that there is value in pro in uh pushing messages to peers that are actually far from you >> because especially if you're the unique producer of a message right because So as as the first publisher of a message >> that gives you an opportunity to start disseminating the message ahead of time in other geographies that are far from you. >> Are you uh so I do think that like the space is really nice. One solution variant here that I'm interested in exploring is bucketing.

>> Yeah. >> Uh by distance. >> Yeah. >> Which obviously has its own problems because it can be games because a peer can always simulate that they're farther further away from you than than they are. And then you can incorporate heristics like um I does the latency match the geoloccation of this IP address for example. It does get complex uh but you could detect scenarios like that. I don't know. It's just kind of like a an idea. >> So maybe just on this but I do think it's interesting and I've tried to through it as well. It's like both. Yeah. I mean if you do it with the IP address for example then it's not very private like you lose some sort of like privacy. I I guess and that's maybe not ideal. But I think the bucketing itself

is also like hard to do like it will be somewhat hard like I don't know exactly how you choose which buckets how many buckets do you have um and things like this and I I do think there is a lot of value in propagating to peers that are far from you uh in the sort of RTD sense of of of the the term but I think what soon somewhat optimizes for sometimes at least um is to send information quickly through beers to reach like nodes that are like sort of far. Um, so it's not like super it's not like you know you pick someone that is far, but like if you have like a good way to like a fast path to nodes that

are like sort of like in further geographical locations. I think that also might be another way to like sort of like think about this instead of having some strong assumptions about like who is where and everything. But I do think it's like interesting like I I I I think we should explore >> for um yeah we should definitely continue discussing this two more like top level thoughts here. >> Yeah. are for large messages. Uh specifically these kinds of um gossip protocols where you need to receive the full message and validate it before you act on it.

>> Yeah. >> Uh the store and forward kind of like protocols >> are quite inefficient for large messages. Um and it's kind of like one of the bottlenecks that we hitting with with larger blocks and larger uh and and higher gas limits which will naturally lead to larger blocks. Um so the solution space that is more appealing is the one of a ratio coded broadcast chunk and so on. So I think that that would be like an entirely different paradigm. >> I I think yeah so it's I mean I agree I do think that this is sort of like agnostic to how large like I basically I I agree that we should like sort of like stop like avoid having large messages generally like I feel like that's just

bad. Um but you know that rule is agnostic to whether that is big or not. It's it's more about like you want to save bandwidth and or you want to like just fast propagated through the network. >> Yeah, it it kind of like the routing strategy is a little bit independent of like the the content of like what you're what you're sending. So so yeah, I agree that like you know the this would be on the routing strictly on the routing. >> Yes. Exactly. Yeah. Yeah. Um then one one other thing is that so right now the the network is actually flat and permissionless >> which [clears throat] means that any node >> that you spin up specifically any CL node even if it's just for testing

>> is going to have the opportunity to join the network and actually influence propagation. >> Yeah. >> And that is one major like limiting factor for us. And there are solutions like for example you know the crypto team worked uh and like some people here I've already discussed with are already aware of this the crypto team uh at EF has worked on has made some progress on proof of validator. >> Yeah. Yeah. >> So that's a primitive that we could potentially use to create to prioritize peers that are actively uh securing the network over observers of the network.

Um so that would like you know constrain the the graph even further. >> Yeah. >> If we were or like even bucket it further because we do still propagate to peers but they just to uh to peers that are not validators. >> Yeah. >> But they would we just depp prioritize them. >> So I think I think that's great. I I I do think it's a bit also different. But what is just like what some of the things you are telling me are just like for me confirming the fact that we need to have like very different buckets and I I feel like the last part could be in the sort of like peer scoring part of things, right? It's more it's more like how do you prioriti prioritize peers you are

connected to? How you do do you even connect them to them in the first place? How do you like sort of like construct a graph of peers around you? And then how do you push? And those things are obviously inter related, but I feel like it's nice conceptually to think about it separately. So you can really like sort of like optimize for both of them and then like look at them together. >> So just to kind of like put some more context there. uh right now implementations are have the uh have a connection table basically and that connection table is actually is is scored linearly based on a number of parameters that >> are uh set by the protocols themselves.

So for example >> um if the more topics you share with another peer on the gossip sub layer the higher the score will be >> right >> for that particular peer things like that. So then you would naturally naturally like score peers that have higher um that have an overlap in your app nets for example. >> Yeah. >> Um over peers that don't. Right. >> But this is a very naive way because uh in reality what you really want to do is segment the connection table into different purposes.

>> Yeah. >> And say I want to find the best peers for block propagation. I want to find the best peers for for column propagation. behind the best beer is for like the for each of the adnets that I'm subscribed to and then even that rotates on on XBox, right? >> Yeah. >> Um so, so yeah, so it's kind of like a more purposeful way of of peering with the network >> and that would like be kind of like the background thread in all of this because then you would be having meshes that are optimized already for a specific purpose.

>> Yeah. with some of which might actually be using a razor coded broadcast like you know in the case of blob blocks in the future. Uh so yeah so then like it's kind of like more thinking more about fine-tuning for the specific scenarios. >> Yeah. Yeah. Scenarios and messages. I I feel like that's also I mean we we've talked about this I think I think a bit but like I think the idea that you already had is like can we like tailor it to a particular type of message? Yeah. Uh >> and I think then yeah just like another thought I guess is like to be careful but not overfeitting.

um like when you tailor it too much then you lose the leanness right like you lose like you actually will like sort of like >> really sort of like put a lot of constraints so you optimize a lot for one particular thing >> but then like yeah then you don't want to do that for like every specific scenario you can think of basically >> and you lose resilience right because the more specific you get then you're overfitting for like some scenario that like you've designed and if something like >> changes Yeah. And things change all the time, right? [laughter] So, yeah, like say adapt.

>> Yeah. >> Yeah. >> Yeah. >> Yes. >> Uh did you run simulations for the network size maybe thousand validators? >> Uh yeah, that was 1500. So that was very small. Uh >> okay. And uh basically like how faster is that uh if we use saloon comparing to just randomly choosing our neighbors uh like do you have this?

>> Yeah. Uh >> I didn't put it in the slide cuz I I went with a progressive approach but this uh this gives you it's so it's not soon but like soon converges to k fastest for latency and bandwidth and so you know you see the sort of like random is this line >> uh and k fastest is this so it's it's not soon like soon can be tuned to like be like K delta Fest but you see like basically um what you see here is like you can go for way lower duplicates

um with very good latency with something that's RT aware compared to anything that's like random or like not not RT aware. So you you do get to like I don't know [clears throat] generally what I saw is like a 40% improvement in terms in terms of like bandwidth efficiency. >> So like lower duplicates for the same sort of like latency or like just better it's just like depends what you choose in the tradeoff spectrum. Again, if you really want to optimize for latency, you set soon with like a very large slack, you get to da fastest and you get like the best latency you can even think of.

You'll just have like a lot of duplicates. Um, but but not more than random. Uh, so it's just like a better rule that can also be tunable, but like yeah, you you get the near optimal latency for uh for the same number of duplicates basically. >> Yeah. Yeah. Just one of the tasks that we have here is that how we can come up with the best top apology for example for signatures aggregation and yeah even speaking about the sub level where we have like a thousand validators and we need basically >> everyone well yeah we kind of have different ideas but one of them is that we need everyone in the subnet to actually observe as many signatures as possible.

Yeah maybe for example 90% that you have here. >> Yeah. Uh, and yeah, I'm particularly curious like how faster we can do this with something like soon comparing to just randomly selecting neighbors for example. >> So we can >> implement it and check. >> Yeah, we can we can implement it actually. Yeah, we have simulation for the aggregation. We use like real world latencies from the database but we also used it.

>> Yeah. Uh >> yeah, I used to >> So yeah, basically maybe we can update I was thinking maybe we can update our simulation so that uh uh before we actually run the simulator, we not just randomly assign neighbors as we do it now. But since we aware about the >> uh latencies, yeah, we can also calculate the most optimal neighbors according to soon and compare. But honestly, I mean we did so many optimizations uh on coip. We tried different ideas and the improvements were so little.

>> Okay. >> Yeah. I which is why I was asking like how much gain do we expect like is it towards like one 2% or 10 20%. >> No it's funny. >> Yeah >> the gain I mean how faster we can reach our goal which is like >> send signatures many to many like everyone observes every signatures. >> Yeah. Um yeah, I think the improvements are actually quite good. U in terms of latency, so pure latency like random doesn't do too bad. It just like spends a lot to get there basically. Um so when you have for example if you have super super small messages like very very

small then like using random and like delta faster so like the latency optimal sort of like strategy won't make a huge difference. Um it's it's more like as soon as you have some messages that you that are like quite large or whatever then >> 3 kilobytes >> what >> 3 kilob 3 kilob yeah I mean it adds up right like it's it's uh if you receive like eight duplicates of each message like of three kilobytes like and that's um it's just like added like added load generally. So I think um yeah I think I think what I'm saying is like you can get to like way lower

duplicates per node with the same latency. Uh and you you can like improve a bit the latency but like the the most important thing you're going to get is like less duplicates because it's going to be smarter about how to like what pier to send it to. And sometimes you the uristic you will use will be like okay for sure that's that pier has received it already so I don't need to send it to him and and so that like you won't send it to that pier and that that peer will like be better off because he will probably have had this message already so he doesn't need to like get a lot of duplicates basically. Um, so yeah, I think I mean if you want like we can try because I I don't think it's

so if you have RTS like if you have like knowledge about RTS in in your in what you're like testing then it's very very easy to implement. It's just like that that rule is so easy. So we we can also like try it out and and test it in like more real life environments and see how it does. But I expect it to to to do quite well. Um, so if I can throw in a suggestion here. Um, so we we're trying to, you know, minimize, at least in my mind, the trade-off here between latency versus bandwidth. You know, we'd like to approach, you know, the the the the best of both worlds. Um and uh to some

approximation that would be given by um optimal solutions to the degree diameter problem which is the problem of you know how do you have a connected graph with as few edges as possible and the and and the and the shortest you know maximum length path right if if that makes sense. So, so um like this is actually um kind of an easy problem if you have a global view of the network and some centralized entity can you know assign the connections between the the the nodes.

So I think I think I think that like a question worth exploring is you know are there rules by which individual node can decide its neighbors uh in such a way that um you know approximations to you know the the best known solutions to the degree diameter problem arise organically out of that. >> That's exactly right like that's how I got into this. It's because I was talking to this guy that was doing optimal transport, which is how to provably move mass across a network of nodes the most in the most efficient way possible. And that's been a problem studied for many many years. And so he was telling me about this. I was like, "Oh, I can develop like two P2P networks

as like a sort of like random thought." And basically, there is this whole like newer because it's math, so it moves slow, but like literature on balanced uh optimal transport in which you can also create and destroy mass um which will map a bit better like to P2P networks because you know you have duplicates and you can also like just like stop sending messages. Um and and then there was this new literature about like so the they call it like a splitting scheme or whatever but like is how to approximate this because like all of this is using a global view like your optimal transport is like you assume is that's part of like the initial assumptions and you never like think of it again. Um but

there was this new work of like how to approximate this sort of like efficient way to do that in using a global view uh with local rules and so that's like a new sort of like unbalanced optimal transport field and I was like oh that that seems to apply to what we are trying to do um and that's where the sort of like downhill idea came from because like you propagate the information like forward or downhill like as a gradient descent and you you don't want to be stuck in local optima or like a local optima like just just like with gradient descent but like it seems to approximate this sort of like global like yeah global view optimization problem. Um so that's why I

liked it. I wrote something about about this there. Um but yeah, I I think it's it's and yeah, one thing I will add is like it's actually not trivial even with a global view to to find like the sort of like very optimum way like people have done it now but it took them like a lot of uh of time to to do that even when you have like a global view of the network which is fun. uh one thing we should do is run these simulations with the size of the current uh Ethereum network.

because that will change the duplicate rate. it will change um yeah the bandwidth efficiency of things because once you actually size it to 15k nodes yeah so that would be times 10 I think if I understood correctly >> um that would like give more opportunity there would be more slack for duplicates to circulate and that is where suppressing duplicates and bringing in some of the total gossips of like techniques uh would really would potentially help here >> so I had simulations with 50k nodes Yeah, >> until I my laptop was stolen and I had to rerun the simulations with 1.5.

But uh but I I originally did have this and you see more drastic effects of >> RT aware rules >> compared to like unaware ones because yes, you have more opportunities to like spread like duplicates all over the network and so that that showed up as well. >> Yeah. Okay, cool. Yeah. Uh, one other thing is that we also have some tooling that computes RT between nodes specifically in the Ethereum network. >> So, it's a more realistic approximation of of the distribution that we have.

>> Nice. Um and then on top of this we're like through the work that I described uh we're trying to analyze the graph so that then we can like put that you we can cross both um like both data sets >> and actually get a understanding of effective RT within the graph that >> yeah that's that's what I want the sort of like better observability was like exactly this like >> please like I want the sort of like global full topology of what we're dealing with plus some like realistic RT distribution and then we can be a bit more confident because otherwise yeah I mean look like if you have the sort of

like random >> um like log normal sort of like network and you optimize for it you end up being completely screwed if like the network changes and you are like into like you go into some like D model distribution or whatever. So yeah. Yeah, for sure. That will be super useful and >> as soon as we have data on this, I'll be happy to also like help like passing through. So >> nice. >> Yeah. Cool. Good. >> Okay, I think that's that's all. Thanks everyone.