Reputation for machines

Transcript

all right welcome everybody good morning um my name is elias and i'll be talking to you about reputation for machines um before we get to the meat of it all a little bit about me i'm the founder of a company called rated we're under rated.network i previously spent about two years working with the teams at bison trails and then a coinbase uh as a protocol specialist looking at all things layer one layer two uh and digging deep in the infrastructure space of of web 3 and before that i spent about three

years at um at a crypto asset fund called the central park capital building up the research and and product functions um and i'm super excited to be here with you today to talk to you about the work that we're doing with rated and the things that we're thinking in our general philosophy about you know what the value of reputation specifically in in in the machine world uh can be today and increasingly more moving forward so as you probably know in like uh justin's presentation ended on a on an interesting note before because there was talk about reputation and marketplace and so reputation is

everywhere right um in in in fact the idea of reputation for machines is really not at all a new thing and and has existed since practically web one and is is pretty deeply ingrained in in fundamental sort of uh internet protocols but you know you you will have seen it on traditional marketplaces things like ebay things like trustpilot things like grubhub and all the delivery services things like uber and what reputation does is is it helps provide

valuable signals to the users of these platforms in these networks as to the quality of things it helps build trust and it helps drive economic activity [Music] all right so i alluded to the idea that uh this is actually the slide should have been rejigged but this is the previous version in any case um i alluded to the idea that reputation for machines is not a new thing and in fact it really isn't right so things like the the dns and it's not dsl this is one of the mistakes in the slide so the dns whitelist and

blacklist and things like pagerank and credit scores have existed for a very long time and particularly page rank in dns whitelists and and blacklists have really to do with machines and the way machines conduct themselves in the context of their network and then in turn it helps machines um understand other machines understand who is their counterpart who are they talking to and what has their history been uh in the context of their existence in that network that we're talking about and more generally if you abstract things a

little bit more highly you can actually like make a distinction on an axis across which we can rank reputation systems from algorithmic to user generated so algorithmic are a little bit slower more first principles what do we want this what do we want to measure what are the behaviors that we want to kind of classify and give quality scores if you will too and kind of come with a high level fixed view of the world and then allow data to kind of pass through that algorithm and generate scores across the uh the index and then there's user generated reputation systems that basically are

more dynamic and are better suited to things that are not you know they're more expressive and expansive like a network of humans that interact with each other and like buy and sell there's like all kinds of um uh points on the spectrum that that people can land um and why is the reputation an interesting thing to be thinking about well because it has benefits so there's scientific studies that have proven over and over again that um more and better reputation in marketplaces for example can lead to people vendors specifically drive more volumes

to increase a premium on the prices that they trust and in fact it's actually also better for the marketplaces themselves so i didn't need to tell you that but it's a it's an interesting uh little stat that uh that we've come across in our work so um pretty well understood insofar as peoples go um but when it comes to reputation for machines yes page rank yes dns but it's not an obvious pairing of the two concepts machines and reputation

but where we're coming from is the idea that machines run this ecosystem that we all operate in whether you're building an application whether you're running a node you exist in a network that is run by machines and these machines are not unboundedly expressive they follow a spec the spec changes over time but they also carry through the agency of their operators the way you tune the machine the way you tune a node the way

you decide to take more or less risk to jack up your performance potentially says something about the way you conduct yourself and since these are games that are played in multiple rounds block by block the more these games are played the more it actually makes sense to think along the lines of reputation for these machines but also for their operators and i'll come back to it towards the end of the talk uh but it's interesting to think about what sort of world we end up living in if these machines that start um talking to one another can only

optimize along vectors that are easily quantifiable like monetary value but we'll get back to this in a little bit so one of the problems that kind of were the impetus for me leaving my my previous life and and starting uh rated was the idea that um in this whole world of reputation and its its components um i i really kept stumbling upon the same issue which i'm i'm sure many of you listening and and hearing the audience will have come across as well uh which is that

you would ask specifically in kind of like the validator operator world which i i spent a bunch of time in and i still do um you would ask n operators what is performance how do you measure it and you would get n different answers so uh i've got an elephant here uh pulled apart in in pieces because that's the allusion to the old saying everybody sees a different part of the animal but in fact there is a whole animal there is an actual elephant in there and that's to say that

these concepts of risk and performance in an environment that has a fixed spec at least for a few rounds of the game and then the spec changes and then it remains fixed for another era and then it changes again but for these periods it's not unboundedly subjective and neither should notions of performance and risk should be so to me that looked a bit like um a sort of market failure if you will lack of coordination among participants in that ecosystem and then i started thinking what

could you build what could you do if actually all these people agreed and then you start thinking and it's actually a series of of interesting things that uh that you can actually build so um this is uh a screen grab from from our discord uh from from jim jim mcdonald who is a legend uh in in the ethereum uh validator operator community and and he like i really like that screenshot because it it captures that uh idea of sort of subjectivity not being unbounded but it actually existing right and

when you're trying to measure performance uh what are you actually trying to measure are you trying to measure how much a validator for example helps the chain move forward or are you trying to measure how much money they're making or are you trying to measure how that validator behaves in comparison to the rest of the network so there's there's a lot there but it's also not unbounded um they are slightly different but they're not wildly different and so um the way we think about

reputation for ethereum validators specifically i should note we've started with ethereum beacon chain we think that the theorem beacon chain and ethereum is the most consequential chain uh that is live at the moment beyond probably any reasonable doubt um and we think that it is the most interesting environment to actually kind of be probing and building this idea of uh machine reputation so the way we're thinking about things but but clearly like this framing is actually transferable to many other things

in many other networks and potentially not even only validators right like sequencer nodes that help uh rollups communicate with the layer one that they draw security from can also have reputation and sequence their sets can have reputation and re-layer nodes can have reputation that help two chains actually communicate to one another and pass messages from one another and it'd be interesting it's interesting to think if you have that thing well contextualized then there's a bunch of stuff that you can build like you know smart relays for example and more efficient ways to actually distribute resources and financial products we'll get that in a bit but for now

let's stick with this formulation right here so how do we think about building that sort of machine reputation layer and we think of this in sort of three distinct categories machine reputation can be composed pretty well upwards from performance externalities that a node produces in the context of its network and then the risk that it bears right and these are again sort of overlapping with with one another but they are

relatively distinct so let's talk about how we go and measure performance and then also i've uh ranked everything from an on chain to off chain uh scale uh that you see on on on the right hand side of the of the slide and and this is basically where the primary data comes from to actually kind of like collect and classify so that you can you can bake it all into algorithms and then indices and so on to actually come up with uh a number that gives you high quality information high density information about what each of these three uh

categories actually how they manifest right so for performance we've uh taken apart the ethereum um beacon chain spec we've looked at the consensus layer spec and then we've basically selected the categories that validators get rewarded out for but we have also left out a few other categories right some some categories that are potentially less consequential and the ratio of how these categories are basically remunerated for performing that job so participating

in a sync committee for example is disproportionately large to a validator actually performing the normal duties which is to attest to their version of of reality at any given round that this this game is played in we have uh underrated.network uh pretty uh comprehensive documentation about how we've we've thought about this uh uh this idea and and and you can you can go and dig into that more but the the crux of the matter is that you know we can we can look at the world in a post hawk fashion and then put everything together into a performance score that evaluates how this validator

how well have they performing their duties over time in a way that also allows us to compose a lot of validator indices which are what you would see on a beacon chain block explorer as kind of the tip of the spear but these indices actually compose upwards to larger operator entities so the idea there is to be able to create or great contextualization of what performance is at the atomic level but then also to be able to compose it via agency right and that agency is who owns these validators who operates

these validators and how have they conducted themselves are they good citizens of the beacon chain or are they not so diligent perhaps citizen of the beacon chain so that's insofar as performance goes and that's also the easiest um element of reputation to capture because it's on chain and it's post hoc and it's more easily available but then when you're thinking along the lines of externalities and risk things get actually a little bit hairier and so when we think about externalities we think about things like how does a node handle

mev for example so for now uh mev is is handled by miners very soon potentially as soon as september this will change forever until further notice and it will be validators that will be handling mvv so validators will actually have multiple choices validators will have a choice to create their own blocks in kind of like traditional spec way to take transactions in a in a first in first out way or do more fancy stuff or just go with you know gas auctions and so on they will have the choice to consume data and take bundles from private

relays like flashbots uh flashbots is methboost and these decisions will actually exert a pretty consequential impact on the application layer will they be front running will they be facilitating users being front run will they be facilitating backgrounding will they be sandwiching folks all these things are actually externalities positive and negative and capturing them is we think a pretty uh interesting and and enrich in in context

element of reputation the other uh parameter is actually risk and and that's where we think most of the information lies actually off-chain there is not a lot you can tell about an operator or a pool um a set of machines in in how much risk they actually carry uh sorry checking out the time in how much risk they actually carry so you got to go look for information in the off chain realm to figure out how is this operator distributed how many data centers are they spreading

their validators on how many clients are they running are they a mono client operation are they running multiple clients such that if a client fails then they have uh failovers and so on so all this to say that we think that if you have this well contextualized layer of reputation you can actually take these nodes from the capital assets that they are software and hardware that produces future cash flows and is bound to risk and elevate them into exploring their financial asset potential with new products like insurance like financial derivatives like

lending and so on and so forth things that you cannot actually put together without this contextual data layer so we have started exploring all these different use cases so if you've been paying attention to [Music] twitter uh these last couple days nexus mutual just announced that they launched a slashing insurance product and downtime insurance product and stakewise was the first ntc to actually download insurance from them we are actually working with nexus mutual as their sort of partner for source of truth and pricing that insurance we've been helping leido

as they consider new onboardings in in their last two rounds among other parameters but but there's an interesting sort of experiment we've been running there we've been working with both implementers of distributed validator technology obl and ssv network which i highly recommend you check both it's one of the next really exciting things that's coming in the beacon chain and then we've been doing different kind of experiments and and white boarding with with the likes of uma on you know what could financial derivatives look like derived from these reputation scores so this is what we look like at the moment it's potentially the least interesting thing about what

we're what we're building but uh it it's there to basically it's under rated.network and it's there to give you an idea of uh what data is actually available and what slicing and dicing and so on but the most interesting thing is uh this api uh link at the top right uh which is which is basically our our uh data accessible via api um which we haven't kind of been very public about but we have already a bunch of people actually tinkering with it and we're getting really high quality feedback so i do encourage you if you are interested in kind of playing with the data and building things off of the data please come to the website go to

the api come to our discord talk to us and and then we'll we'll do everything that we can to help you actually uh uh figure out the context that you need um so quick few things about what's coming next for um excuse me what's coming next for us um we are in the process of kind of moving out of easier and going into v1 a big part of of this effort specifically with the merge coming is actually contextualizing the wealth of sort of interplay of interactions between the consensus layer and the execution layer things like you know how to how do validators handle transaction fees how do they handle mev classifying some of that mev and so on so there's like a big scope of work that we're

pursuing there and then this will also come with a refresh of the network explorer just making it a little bit more expansive and expressive and and the same thing with the api and we are also hiring so if any of you are interested in learning more and even potentially working with us please do get in touch i'm at eliasimos on twitter we are at rated w3b on twitter or you can just uh email us at join rated dot network we'd love to talk to you we're looking for people that like really resonate with what i've just been talking about and really see kind of like the value and

transformational potential that this whole concept has in this for all intents and purposes um still pretty nascent and yes we are still early there's always summer we're early in so this is us uh you can find us in rated.network and at rated w3b on on twitter thank you very much uh and let's move to q a [Applause] hi thanks for your presentation um i was

wondering you mentioned a bit about mev invalidators how would you prove that validator is not extracting mev good question you'd probably look into how they prioritize transactions and so there is a very large scope probably if you know the adoption of things like meth boost is as is being communicated so to the tune of 80 of of blocks that are being produced that are coming from from private relays you can possibly be listening to both sides so you see the print on chain but then you're also listening to what's happening on on the relay side and then that's like okay you

know which blocks have mev in them because they came from that relay then i guess you take a smaller sample than the 20 that remains and you're trying to actually ascertain what's happening there is it um are they actually is the builder of the block which is probably the proposer if it's not coming from like a private relay and so on are they following the spec and are they doing kind of like the vanilla way of block building which is i guess prioritizing by uh by gas or are they doing something fancier right it's it's it's highly contextual and it will be a process and as you know mev kind of uh it becomes increasingly

more complex and there's new strategies that are coming along and and you can't possibly capture everything um but you can capture a fair amount and so that's kind of a general thing that we we believe that you know radical changes are awesome but they're also not entirely realistic right it's in in something that is very consequential and impactful even sort of marginal changes to be able to say figure out eighty percent of how many v is if there's a mev in a block or or not or if you're able to not figure out exactly like the full value of mvv in a block but let's say the baseline mev so at minimum there is this much uh we think that these are like positive steps in

the right direction and can unlock things already as they are okay thank you elias nice presentation thank you um so um for example with lido um what does like the data that you're providing them like look like is it just like a score based on all the validators that they're looking at and then they just take like the best ones and delegate to those people or like what does that use case look like and process yep great question um so this goes back to to kind of the usefulness of the of the front end as you can see there at the top we have

a participation rate inclusion delay and and correctness which are i'm not going to tire you with what each of those means but you can go and read uh about them uh in in our docs and these sort of like bubble up into uh the effectiveness rating that you see there which is basically our attempt to capture uh how well a validator is performing according to spec uh and then also actually project uh rewards forward with something that doesn't have randomness embedded in it right so if you auto regress effectively uh a validator's past performance exclusively looking at rewards and draw the linear regression forward you'll get like results that are slightly skewed because

there are these kind of party rounds effectively if you will that are valid there can participate in they can get like a really big reward but that that isn't necessarily reflective of their sort of long-term potential right and so we've kind of like removed this 15 20 of of randomness from the way we sort of view the world and we've come up with a score that that captures um their sort of performance in the context of of the network and so um lidar has been looking at you know things like uh penalties that validators have accrued we have i believe there is over here there's a contextualization of that and also the effectiveness score right and they have like a bunch of

other um parameters that they're they're looking at like the off chain stuff that we we talked about but this is kind of like another layer of you know screening and saturday checking and so on that they uh they can they can bake into their decision making about allowing more people into the validator set cool thanks cool we're done thanks we can sorry grab some time later outside thanks everybody foreign