Collin Myers & Oisín Kyne: Obol Network - Distributed Validator Technology (DVT) #497

Transcript

welcome to epicenter the show which talks about the Technologies projects and people driving decentralization and the blockchain revolution I'm Felix and I'm here with mayor Roy today today we're speaking with Colin Myers and ocean kind who are the co-founders of oboe Labs oval is building software to enable distributed operation of validators starting on ethereum hi ocean and high Colin and welcome to epicenter hey guys thanks for having us pleasure to be here yeah so uh both of you actually were in touch for like a long time you've both been in the space and in in staking especially uh for a while so you know as is customary on an epicenter we'd love

to uh first get an introduction to how you how you got into crypto and and how you got to where you are today uh maybe Colin you can you can start yeah cool yeah thanks thanks for having us again um I have been watching and observing epicenter from a distance and always wondered when we'd get our shot um so yeah happy happy finally came definitely on the longer end of the spectrum uh based on my expectations set but yeah happening finally made it happen uh yeah so quick quick background on me actually non-developer by background uh used to work on Wall Street worked at a variety of different banks um mostly in like credit and debt and

got involved with ethereum Community uh 2017 I was living in New York and you know lucky for me there was consensus and there was Joe Lupine and there was meetups and there was different stuff happening and at this point in my life like 2017 I was trying to spend a lot of time you know trying to work at a hedge fund or a private Equity Firm or actually honestly just totally lost as to like what I was supposed to do um and there was two things going on at that point in time one of them was wework and the other one was crypto in New York and those were like the two main things that everyone was buzzing about and work worked and trying to get a job there but also spent most of my time on ethereum

um 2017 I met Joe living in a Meetup I was inspired to quit my job so I did and then I joined consensus in 2018 and I worked there for about three and a half years um at consensus was able to work on a lot of different Technologies even like space and blockchain um and some crazy stuff like that but primarily focused on staking um I started a project called token Foundry uh which was like an early mover in the space to helping networks launch and trying to do compliant token launches um Turned that project into a project called activate um which was recently sunset by codify something we actually partnered with on course one in the early days um and throughout those time periods I I

spent a lot of independent time contributing to eth2 was involved in early working groups after Devcon 4 and I primarily at the beginning focused on the economics of ek2 they were not very well understood and my job was to help enable the understanding of the economics for your at-home validators for your larger scale validators your exchanges people like you know coinbase and consensus and these different types of actors those initial work streams led to a lot of different enablement projects around the Genesis event so the the majority of my time was spent around the Genesis event and enabling that to take place um while at consensus we built the EQ

launch pad which is the sole individual place where you're at home validators interact with the deposit contract and spent quite a bit of time with the client teams and helped them with user feedback and ultimately all of these research projects culminated in what today is now DBT um 2019 uh Docker fights and Carl B started a research group focused on trust minimize staking and marshmead and myself were the first to to come in to help enable that effort um got completely sucked down the rabbit hole on DVT um thought it was uh world changing technology three years ago no one had any idea what we were talking about um it was uh kind of grossly underfunded

um as an effort and ultimately the only way that we found proper funding to turn DBT into a project was by ocean and I both quitting our jobs respectively and going out and you know raising capital on our own and finishing the effort and putting the group of people together to make the technology a reality so yeah that's me um thanks for having me on I'm an Heath Maxi loving Cosmos these days though um but yeah really excited about it um and then yeah to give my background I'm a software developer by background and in fact I'm the fortunate person who's like second generation internet entrepreneur my parents have run web companies most of my life um so yeah I would have grown up around

kind of original website building web design in the like late 90s early 90s um so I kind of SAS companies built in the kind of nowadays and 2010s um out of college I worked as a consultant and kind of discovered crypto in 2017 and got my first kind of full-time role in 2018 when consensus opened an office in Ireland um I did two years there in their tokenization Department we did some project Theory tokenized Securities for French real estate on mainnet back in 2019 before that was prohibitively expensive to do on mainnet and I left consensus in March 2020 and I Incorporated myself as a self-employed kind of contractor I got picked up by

black demon to do either to research for them back in 2018 when the original E2 specs came out my I wrote an article that was effectively quite critical of them at the time it was 1 500 ether minimum and slashing was fairly severe and I was like really do because this was going to be safer makes sense and then yeah I kind of kept an eye and it got involved with block demon um had been building it along and ended up building out their E2 stack for them by Genesis and I'd say shortly after Genesis I was first introduced to the idea of trustless validators by getting an invite to this uh trustless validated Community Hall that Colin and Mara were running and at the time I was kind of

agonizing over having no ability to run a backup validator or do anything when one failed and I was introduced to this technology and I was like oh it's high availability for validators this is definitely going to be a thing everything in web 2 does this and I was kind of an immediate convert but yeah I think Colin realizes not very many other people were um and yeah we it took me maybe a few more months of kind of being involved I was trying to make an nft issuance kind of Stack I did some work trying to issue like rugby video nfts with some kind of like famous rugby players that didn't really go to plan and then um someone once approached me and asked me to make an erc20 token for to represent and steak and I kind of counter pitched them and try to make

like nfts to represent validators um and then yeah it I wasn't really very good at selling any of this idea I had a product but I had no business uh Acumen and I realized that I wasn't really very cut out for the whole CEO role so I reached out to one of the only business people I knew in the space who was Colin because a couple of people had been kind of pushing my me his Direction and I was talking to about what I was doing he showed me what he was working on and what he wanted to do with obol took me a few weeks to get convinced that uh it was not too hard to attempt and yeah I think we kind of started around April 2021 together and here we are maybe two odd years later but more awesome yeah that's a rich history you

both have and and thanks for going so deep into it I think yeah we personally also have worked a little bit on high availability validation so like we're we're both very excited about this episode uh so yeah Keen to get into it right so I guess maybe we can start there we already mentioned DVT a bit maybe can you explain you know what what is distributed validator technology yeah cool I'll I'll give kind of the more macro perspective as to like the difference between a normal validator and then um oh she can get a bit more into like technical architecture and deployment and stuff so um today a validator consists of three pieces um it is a public private key pair uh it is a machine and it is an agent an agent

we look at is an individual or an entity so it's three things and all of those are super monolithic and that's just a validator stack of today uh from an ethereum perspective what DBT enables is a more modular validator stack and now what happens post DDT is you have key shares not just one public private keeper you have a collection of key shares as many as you'd like to create based on your trust you know properties essentially the next piece is that those key shares then can go on multiple machines not just one machine but multiple machines and then third and lastly this validator can be run by a group of people or a group of entities so it takes your

modular validator today super monolithic and it takes it to the more modular version where you can have multiple key shares it increases your security you can have multiple machines it increases your availability and you can have multiple people or entities therefore it increases kind of your game theory or decreases the chances of Byzantine behavior for that validator and I'll give it to oshin and you can go more into kind of the technical talk yeah I think in epicenter podcast is maybe one of the only places where I can talk about the like nerdier details of this rather than the high level idea but yeah I think one of the interesting things about it or what you know makes

these distributed validators more possible than they might otherwise be is the cryptography that ethereum to I should we should probably stop saying they're ready to uses proof of stake ethereum and that's the BLS signature scheme what's fancy about the BLS signature scheme is it's one of the first like elliptic curve signature schemes where if you have you know independent signatures all for the same hash you can actually like add them together in a low trust environment you don't need the original private key to do so so what happens as Colin alluded to is you know you want to set up a distributed validator the first thing you probably do is a distributed key generation unless you want to just spit a normal key locally but it's better if you kind of do one with a dkg then at

the end of that process there's you know let's call it four machines to keep it easy and each with a piece of the private key on it and you have a piece of software in the middle of your valid data stack between your like consensus client and your validator client and it these four nodes can you connect to one another over like TCP and they more or less play a little consensus game to decide on what they're going to sign every time that there's a validator Duty coming up they you know play a little consensus game pick this is the hash we're going to sign and then every validator client is given the exact same you know hash to sign they all you know check it for slashing rules do all the usual things what's nice is everyone gets to keep their own private Key

Management um you know the oval software is just going to read only is middleware it doesn't actually have the power to sign anything um so all of the independent validators check that nothing's flashable once they're happy they sign it with their piece of the private key and they broadcast it to what it looks like they're a consensus client and then in the middle there our software like intercepts them shares them around with the other machines and once Annie of in this instance like a four node cluster once three of them like this three of the signatures are together you can combine them into a full valid signature for the validator and you can send that onwards to the network and what's neat about that is you don't need all four Sigma managers you actually can have fall tolerance and this is kind of one of the super important things about high

availability is if you put you know a validator on four machines but you need all four of them to be online you're more brittle than you would otherwise be but if you put it in four machines and you know only need three to be online this is where things really start to change and you can have high uptime you can do rolling restarts you can replace Hardware that fails without downtime and you know all of those good things and are kind of unlocked with distributed validators so from a user perspective if I'm a Staker generally the options I have today is I can um I can either put it into a liquid staking protocol the LIE Dome would be the biggest of them I could go to a

specialized validator shop like a block demon or chorus and I could spend about a P2P when I spin up my evaluators there or if I'm usually technically competent then then I can run my own validator I'm actually curious kind of in these three three kind of segments let's imagine that being liquid sticking as one segment the other being the specialized validator that's like hosting validators for you if you have 30 tweets third segment being [Music] um you want to run your validators at home um how would these three segments kind of use this technology and what difference would it make for

them yeah it's a that's a good question so I'll I'll talk a little bit about like our adoption path so far um the earliest adoption of opal has been at home validators they're the first ones to have put it on mainnet uh we put our first ones you know on mainnet between core team members all running at home um and now kind of the adoption evolution is now getting into your hosted person uh and you know your liquid staking pool liquid taking pools were kind of the earliest uh eye doctors of the idea and supporters of it being in their roadmap um so the I'll start with the at-home validator uh which is quite interesting

so today the at-home validator for example like ocean runs runs one out of his house and like we travel a lot today for for for like purposes of mobile um and it's always it's like kind of chronically offline it's kind of this this thing that you have to worry about and you can't really have peace of mind for being an at-home validator today uh which is something that like DVT can unlock for you right you could have your at-home machine uh you can run backups inside of the cloud in a non-slashable way uh so that you can ensure you can kind of live your normal life um the other side of the at-home validator is the Persona type of I don't have 32 eighth but I trust myself to run my own machine and there's actually like

quite a lot of those people uh and they can use opal to come together to like create their own shared validator because they may not be interested in taking on the risk of the pool and you know they may not need a liquid stake token but they don't have 32e and that's kind of this squad staking concept which which we've seen uh honestly take off quite a bit in the at-home validator category which is you know a whole group of people who's like you know some of my ethas in a pool the rest of it I just want to run myself but I don't have a full validator's word uh which has been quite interesting so yeah it's peace of mind but then that second user group is actually where we think most of the tail end adoption for DVT will be in like 10

to 15 years is like enabling just groups of people to get together who run their own machines and giving each other fault tolerance and like a human to human manner so yeah a lot of early adoption there but the middle adoption of DBT won't be there in my opinion they're like the early enthusiasts they're the ones to help set us up our primary duty is to figure out how to give the at-home validator more tools to get more of them online and now we're bridging more into the hosted and like liquids taking full category so now I'll get into those two different types of users and you know why it's beneficial for them um for your liquid staking pool it is actually the most Innovative use case of DVT today uh because most of the pools

are using it for its Byzantine properties uh which is quite cool so the example I like to give is let's just you know use a hypothetical liquid staking pool today there's 10 validators inside of this liquid staking pool all of them are supplying their own you know Keys uh to the system keys that they create keys that they manage and keys that they run on their machines again in a very monolithic environment and the reality is is that like in each in in this liquid like this hypothetical liquid staking pool each of those validators would have like a certain amount of stake that they're responsible for uh and that person can actually self slash they can be you know they can sabotage they can act malicious they can't take

the funds right but they can shut the machines off and they can like bring penalty to the greater pool if you will when most of these Pools use social economics so it does mean that like it could take you know it'll it'll hurt every single participant in the pool uh if one of these validators defects and it doesn't even need to be malicious right it could also be a bad day at the office where like you know all your servers crash and everything goes down or it could even be catastrophic right like you have a rogue employee who steals the keys and you know there's a variety of different things that can happen inside of that construct so when DBT it enables is that that group of 10 people can now just share one validator which like creates this whole new defect

Game Theory between the different operators in that cluster it gives fault tolerance and it makes it essentially I hate to use the word impossible but it makes it very very difficult for AP1 validator in that group to do anything malicious or defective to that pool um so that again like that's availability and slashing protection reasons and utilizing it for the fact that there's a consensus mechanism and it prevents Byzantine Behavior Uh so it's one of its true you know core Primitives the other reasons that they're using it is not only for that but they're using it as well to decentralize their validator set so if you're going to build a liquid staking

pool that wants to be very decentralized then you're going to have to invite uh call it middle to lower to sub-tier validators to like decentralize the validator set over time and you need to do so in a way that protects the pool especially if it's an existing pool and most of the pools today like migrating DBT are existing tools and there's lots of money inside of all of them um so if you are going to let new validators in you need to let them enter the pool in a manner that doesn't hurt the pool and DBT is a great way to do that because you can just build some shared validators you can partner you know three highly proficient people with one newcomer and then you know you have some fault tolerance there's some give there if they make an error Without

Really harming the pool um so yeah the decentralization of the validator set is also a very interesting one for pools and the use of it for its Byzantine properties um your hosted service provider is actually someone that we thought would come later in the adoption cycle however they've come in quite quickly um and it's a factor to do with uh the centralized staking product or the hosted product is it was the first product in the industry uh it's been competitive for a number of years now it's like entering its fairly normal maturation cycle of compressed margins and increased cost thoughts and now those products like need to mature

themselves in a way where they decrease their costs but they improve their security as being a centralized provider like requires sock 2 soccer 1 compliance these things help to kind of institutionalize yourself and most of these people have to offer some type of off-chain insurance mechanism which is quite costly to do or or you just have to be pretty well you know bankrolled so for for these users their maturation and growth is meeting DVT kind of in the middle uh and they're experimenting with it to help scale their operations in a way where they can decrease the amount of machines they need for a certain amount of each while also increasing the security profile of that setup we've seen a lot of interesting things

recently in the insurance and reinsurance topic around DBT offerings taking Insurance it has been difficult historically because you know the definition of a bad day at the office is losing all your ether potentially uh but that was just at the beginning and now we're as a community DBT being one of those Technologies building things that like alleviate a worst case scenario of like you know losing all your eats so insurance providers have been interested in kind of including DBT into their criteria of providing Insurance because it gives them more Assurance uh that like you know you can't lose all of it it's it's done an absolute loss

situation so you know no no insurer is going to ensure something that's an absolute lost situation uh and Technologies like DVT have now come in and helped like alleviate that so yeah those are the three core user groups I I think the fourth one I'll mention um is kind of defy and how D5 has thought about uh using DBT so like in most D5 projects today you're like taking an asset and you're wrapping it or you're taking a collection of assets and you're pulling them together and uh you may produce a stable coin from it right or you may produce other streams of yield or income off of a collection of tokens uh the collection of tokens that people are using today are LST tokens liquid state tokens uh and

they're doing a variety of different things with them um those products since there's a product built on top of it it's better for those products that those liquid state tokens are as de-risked as possible and now people are looking at what is their risk criteria and mostly they haven't been looked at the only risk criteria that an LST token has been looked at to date is liquidity because that was the biggest risk right how liquid is this thing like what does that look like but but now it's more or less like okay what are the potential penalization properties of these lsts and under the hood like what types of security protocols and parameters are they using for it so now we've seen D5 who has like its own interpretation of brisk honestly probably a more mature

interpretation of risk than the validating community and now dbts kind of come into their Spectrum over the past call it month or two uh which has been very interesting interesting for for them to want to include it in their ecosystem and and to learn more about it one of the downsides of a DVT like setup is that it adds extra cost meaning what's the what's the other case let's say you have a hosted service provider and uh they're running a validator on some some Cloud probably Amazon or Google and they have an engineer there for uptime and they're running it on a

single machine they won't have a 100 uptime but they will get to somewhere higher than 99 percent uptime just on on on on that setup now when a when a DVT setup comes into the picture um you're running like three or four machines and usually spread across different parties and you're also adding some kind of key setup and key teardown cost right like probably the setup of these systems uh will need some extra extra work it's one of these cases where you are adding fault tolerance um but you're adding cost and sometimes the custom customer cannot perceive the extra fault tolerance

meaning that they'll go to some kind of log explorer that will say hey what are the returns of the evaluators and the evaluator with 99 up percent uptime running on one machine their return will be X and then they'll see the DVD set up and it might be marginally better like X plus you know 0.1 percent or 0.05 percent how do you kind of address that uh address that issue and uh do you see that issue in in practice and uh what does it mean for the different parties and who might adopt DVD I love this question um because it actually pushed back a

little on an increasing cost you're writing that increases Hardware but depending on what people have done beforehand it doesn't necessarily increase their cost and what I'm kind of getting at and you kind of talked about it in the cloud is um originally without distributed validators and the real problem was that there was more or less no safe way to run it back up um the only kind of option you had was do some sort of monitoring game and pray that you're monitoring is you know I'm perfectly reliable and there's not a scenario where you turn on two machines the same private key at the same time um and the problem would not be able to run a backup is um it can be very hard to recover from a failure and particularly if you want to use something like a bare metal machine

and like a data center somewhere that's usually one of the more low-cost ways to run a server and the problem is if you do put you know particularly a large amount of private keys on a bare metal server like that and it just goes off line which is regular occurrence will probably happen at least once or twice a year and you just kind of have to sit in your hands there's more or less nothing safe you can do at that point well you can try and bring up a backup and hope to shut down the backup before the primary comes back or you can ring up the data center and ask them to kind of go and like pull the plug out the wall um but this not being able to like run a backup um has changed a lot how people have you know built their architectures in the

early years um most of these centralized I say most that's not quite true a lot of centralized providers at least started running in the cloud um particularly because then if you do have a failure of your machine you can kind of detach the disk you know it's all in software you have a lot of power to kind of actually recover things that you don't have when it's just a server sitting somewhere um but the problem with that and you kind of alluded to the fact that they might run one machine a lot of people they'll have at least a few servers in the cloud normally they'll have kind of at least two that runs kind of Els and CLS um which is kind of the the meat of it then they'll have a machine with their validator on it and plausibly they'll have some more with a key

manager on it as well so a lot of people like kind of Enterprise taking validators you're talking kind of three to seven Cloud servers um and Cloud servers are much more expensive than a bare metal server rule of thumb think about 10x more including one of the real kickers is bandwidth costs which I'm sure you guys are familiar with with egress but that can often take up 30 percent of the kind of operating cost of a cloud validator so you start to pile on egress multiple cloud machines and so on and it really starts to end up kind of costly and that's kind of way one way to have you know high uptime without distributed

validators is to kind of put it all in the cloud have a bit of failover and do things that way the other option that somebody to go down is they kind of do the hardware route where they'll buy a particular server that has dual CPU sockets so they have two CPUs they have a raid array of disks so if even one disk fails no problem they'll have dual networking cards so you know if there's problems there they'll survive maybe dual psus but these PCS cost in the tens of thousands of dollars and they're not particularly cheap so either you know trying to get kind of fall tolerance by using cloud services or fault tolerance by using really expensive Hardware both of these are you know not as cheap as you expect a lot of you know some of

these Stacks if you're running seven machines in the cloud you wouldn't feel spending a couple thousand maybe three four thousand a month for something like that and if you then have distributed validators um and you can have private Keys like split into groups it opens up the ability to more safely put validators on bare metal machines because if one of your four machines dies no big problem it'll stay online you can kind of wait for it to come back if it doesn't come back you can bring up another you know partial and another node it's not going to cause a slashing it doesn't have a full key on its own all of these machines are doing consensus beforehand so kind of what we talked a lot of the centralized providers that are like how could you know distributed validators possibly not increase my cost

um we tell them that you know with software fault tolerance you can move away from like cloud-based Solutions you can move away from having like extremely you know fancy Hardware based solutions to have high uptime and instead you can move towards commodity bare metal and use just fault tolerance to do it so instead of having you know four to seven 400 bucks a month nodes that run in the cloud you can have you know 100 a month bare metal machines and your full stack and cost 400 a month instead of 4K a month um and yeah the last kicker that is can you do it at scale with high load and we're you know running a lot of good tests in that regard seeing if we can do this at like you know thousands of keys as well to to improve margins but yeah

naively in the scenario yes you're running more than one node but bear the home Staker most Enterprises most you know operators within the LSPs they're not running a single machine unless they're you know doing something with very fancy Hardware they're probably running at least a few machines so distributed validators and having fault tolerance allows them to kind of yeah use cheaper servers less you know not over over provision them as much and we're reasonably comfortable buyer you know the people that are already running on bare metal have a very low cost basis there's a very large amount of validator operators that we speak to that DVT actually will let them reduce their costs even if they are running more Hardware I think your question mayor as well kind of presents the

the cell of DVT you know today because look at the end of the day it's a security protocol it's not a yield protocol which is like what everyone wants everything to be right everyone wants to hear like I'm not using your thing unless it makes me more money right um and like for us the fact of the matter is is that it's a security protocol today and maybe there's ways that it can get a validator to become more profitable but it's not this Lottery mechanism that you download onto your node and then one day you see 200 each sitting in your account it's it's not one of those things um so getting people to adopt it yes has taken quite some time um but there's a reason for that because

it is a security protocol so um it's supposed to give you other things uh that benefit you in the event that there's an increase cost for it however we we think to Ocean's points that uh it's not only a benefit to them on the security front but it will actually probably save them money on a relative basis today and then in the future time periods up to us to kind of standardize DBT as a configuration and what we like to use internally is that we've been calling this like the opal inside strategy like similar to Intel right like what intel was able to do is like basically standardize it's

the inclusion of their chip into like everything and then the machines and the software and everyone else you know were the competitive market uh we think getting like oval on the inside of these things is far more where it sits because it's a security protocol so like how do you position a security protocol in a Time wandering a bear market and two in an industry where like you know everyone's looking for yield right amazing I think yeah we talked about a little bit now the sort of cost from the operator side I guess there's also Obel as this middleware that obviously you guys are developing you're spending a lot of resources on on developing this and you also raise money so

um there's also like an expectation obviously of this having some sort of business model maybe if whatever you can share in terms of like how do you imagine like sort of uber to be adopted is it like The Operators that would pay to to run over uh is there like some other uh models that you've thought of um to to utilize or yeah yes yeah it's a good question so I'll tell like a little story here and have been fortunate enough to observe like how the eth1 and eth2 client teams like were developed and hardened and staffed and funded uh and that's really kind of like the core of like how middlewares

and opal and others you know what types of monetary streams they'll be able to like take on and work with so like the early Foundation of ethereum was you know there's five client teams I think maybe six actually on the on the POS side and then I think maybe three or four on the one side but anyways these are all kind of you know privately bankrolled now uh they're well funded it took years to get to this they were funded by the es and other like you know large donors and that software uh is free forever right so it's virally free it's the primary network access to the ethereum network it has basic functionality but it's going to be free forever and

now like you know Danny Ryan uses the words like ossification and crystallization and really what that means is that like the amount of innovation that takes place at that core client level and funding is going to lessen and lessen over time at new Innovative space needs to be opened elsewhere and now the new like the new Innovative space that's being opened elsewhere is being enabled in this middleware layer so now we have this whole new middleware market uh where there'll be a variety of different protocols who are coming to add complementary software to the core ethereum stack and that middleware layer today has private funding but tomorrow

can cannot have private funding forever right there's like uh there has to be a way for these middleware protocols to maintain themselves but also continue the ethos of giving back and having regenerative economics so today we think that optimism has taken that responsibility it is a responsibility at the end of the day if you're building at this level is to try to figure out how you can make circular economics of some sort um so the way that optimism has like built their Network and started their ecosystem having a fee which comes in that is then donated retroactively as a community to other projects that support the core ethereum staking stack so this is you know people like a protocol Guild

these are people like beef taker this is git coin there's a variety of varieties like uh projects that require funding to maintain themselves on a different level of being open source and opal and DBT sits at this new low enough layer that like we believe it's our responsibility to utilize retroactive public goods in some sort of way uh is the primary way that we start the economic machine uh where it goes after that you know is to be determined um but there's a way for us to kind of use that economic model with every validator type which is also the important thing that economic model works with your at-home validator the economic model works with your hosted person and it also works with your

liquid staking tool and what other other future you know topic or use case that comes about so yeah today we're most focused on what is the version of retroactive public goods that works for opal um right now we're like learning and figuring this out through this main net adoption time period you know oboe's totally free today DBT is completely free you can go and you know play with it do whatever you want um but like tomorrow how do we Bridge into like a circular economic system uh and that's our biggest Focus today I'm I'm actually curious if if you think the ideal place to house this kind of middleware Stack I mean the specifications of it might be

actually something like the ethereum foundation itself and then the client teams actually implement this middleware as part of their as part of their code basis yeah so this is this is Oppo V2 today we are on our roadmap to B1 at the moment where we're about 60 70 percent to be one but we've been working on a parallel work stream which is oval B2 and oval V2 um like further protocol eyes DVT by turning it into a specification um so we've actually partnered with nethermind uh for for them to to be the second core development team for the

opal network uh and they will be leading and partnering with the oboe Labs current core development team for the oval Network to work on the research and specification uh of the opal V2 protocol uh and we are pushing towards a spec driven environment where we hope to incentivize a variety of different implementations right for different people's use cases um and the reason that we're also doing that is to kind of the prior story that I told which is you need to be as close to the base layer from a variety of different perspectives as possible one of them is roadmap you have to keep your roadmap on you can't get in front of the motherships roadmap you got to kind of hang out in this Goldilocks time period

and yeah those are those are the most important things that you have to follow and kind of stay on top of yeah there's one extra thing I want to add actually about the middleware side or as you said like why don't the client teams implement it you know directly this is something we looked at ourselves for quite a while and it's actually why I touched on BLS signatures being aggregatable being so important and I'll talk about it kind of by way of talking about Mev boost and before that even move get um so for those of you that aren't familiar math boost is now the product like run by flashbots that allows you know people that want to get you know inclusion into blocks without getting

front run um they kind of talk to validated through the software but before it was called Mev boost it was called mevket and it was just like a slight modification to the original get code base and they were like highly successful in their roulette more or less all of the ethereum miners back in the day were running Mev Geth to the point that people are concerned with north of 90 of clients and the ethereum foundation were kind of concerned about this they were like okay if there's you know anything goes wrong with the software you know almost everyone is running it so when it came time to move towards um eth2 and Reinventing it to figure out how it works in this new paradigm with

validators one of the best changes they made was to re-architect nevgat instead of it being like a forked client it's an optional middleware or more accurately a sidecar that all of the clients can add and rather than it being you know a feature that only one specific client has and the others don't it was something that they could all opt into and this you know massively de-risked it and allowed it to be more accessible and didn't you know prevent client diversity in any way and we had kind of a similar like issue when we were developing distributed validators as well the easiest kind of way to make an MVP is to make a standalone validator client that has you know arbitrary power to sign you can build your distributed system to go

and run validators this way whereas um yeah the kind of problem here is either you have you know one client that's super successful and it works or you respect that everyone has to implement but if you do that you kind of really slow your roadmap you kind of have to get everyone to March along at the same time and you really don't have much optionality if a new better version comes along people can't switch to it very easily so in the interest of you know not you know causing harm or trying to do good when it comes to building distributed validators we realize that we could also build distributed validation as a middleware because of BLS signatures being aggregatable so um right now all of the existing

validator clients they can add our software into their stack and become a distributed validator more or less with no changes and this is super beneficial because it does allow you know you're not one client or you know replacing the model or if there is you know a better spec comes out or you know overall you know goes to zero in the morning a new one can come along all the clients can just put in a new middleware and it's much more modular and it's much more like fall tolerant in that regards that you know if something goes wrong you know no big deal people can kind of pivot and this you know design idea is why we kind of went towards the middleware root and it's you know something that's kind of served as well in that regard but the last leg of it is

you know just one implementation of the middleware is you know it's efficient from a safety perspective but there could still be aliveness risk if you know the thing has a bug and it goes Offline that could be a problem so they kind of further like the journeys after you build it once get some adoption prove that it works then it starts to become more sensible to protocolize it make a spec have a couple implementations but even if there is just one it is still modular replaceable and is this you know nice thing that if a better one comes along no big deal you know it can be swapped out so um I think that is kind of an important design decision versus why isn't this you know a spec that all of the clients implemented because you can iterate

faster you can try more things you can have more optionality and that's kind of the way we've designed the software so far okay cool I guess we also wanted to sort of talk about how ubo interacts with other middlewares in the ethereum stack so you already mentioned like math boost here I guess that that might be a good place to start where like in my imagination right like I guess now we run this overall cluster like all of these nodes in the cluster also run nav boost do they have to make basically consensus on in terms of like what sort of block you except there or can you talk a little bit about how how this interaction works yeah

um so it's relatively straightforward because MAV Booth talks to the consensus clients whereas we're almost a little layer lower down talking to kind of validate our clients and you ask a good question about do you have to kind of come to consensus on what's provided um Mev boost is a bit different to you know a normal block proposal in that um the fear is that if you have um you know a block that's extracting a lot of Mev that you know um yeah taking like an ad size reward um the Searcher doesn't want to show the proposer exactly the follow block because then the pros will be like oh thanks for the alpha I'm just going to

rewrite this to send it to my address and I'm just going to propose it and this was actually one of the reasons around the kind of redesign of booth now when it moved towards proof of stake was if we want this to be available to every proposer we need it to be low trust because you know in the eat one world there was only a handful of miners so you could kind of trust them whereas in you know each two the hope is that there's a huge amount of validators so you do this does need to be low trust so what actually happens is the relay in the situation is actually the kind of trusted parody they have the full block they know you know what it is and they promise you know not to rug or like undermine the Searchers and and you

know screw with them so they just provide a hash like or like the header of a block to the actual distributed validator so at that point yes the nodes um do come to consensus on you know what one to pick but there is you know there's not too much that can go wrong there's not a full block there they can't reorder transactions they can just say hey you know this is a block editor it's going to pay us this reward are we cool with this and everyone says yep sign it and sends it back to the relay who then appends the the real block and sends it onwards um so yeah that would be kind of integration works and do we want to talk about some of the other ones Maybe yeah I guess the other thing we want to

talk about here is also like sort of a trend that's uh kind of coming up in ethereum right like restaking and eyeing layer like sort of reutilizing your collateral or your ethereum validators to offer additional services like let's say maybe Oracle or whatnot and I guess that's also like first of all it's very hyped I guess but also seems to interact with with Uber on the front that you know um yeah how how could like restaking Services be offered through Uber like I guess that that would be something that I'm personally actually very interested in for sure yeah it's a great question and something we're chatting quite a lot about at the

moment these days so um I talk about so we're talking to them for the most part about a project called eigenlayer who's introduced this idea of restaking and as far as how old fits into the equation there's kind of two ways that we fit in first as a Staker which is you know the the kind of Crux these are the people that are you know um pointing the withdrawal address of their validator at a you know an eigenlayer smart contract and opting in to be the restakers like yes you know we will provide Economic Security for some extra I think additional validated Services what they're calling that the other thing and from a staking perspective we think

distributor validated mobile is super important because if you are trying to you know Sell Your restaking solution to these other services that are you know buying Economic Security from you um you want to be extremely sure that the stake underneath you is safe and secure because if something goes wrong and they all get slashed your Economic Security kind of disappears um technically you know they'll be able to you know charge the person that gets slashed and they'll get penalized for it but if you're you know maybe running an oracle or something and depending on this if there's a mass slashing under you um all the validators get exited so your Economic Security just kind of

disappears on you all of a sudden and that's you know something that you can't really or you really don't want to happen if you're trying to you know if you're uh you know additional validating service looking for you know Economic Security you want to be sure that the validated need is Rock Solid so this is where distributed validated is really kind of odd benefit for the kind of restaking Paradigm it's like yes you know this stake is run by groups of people the odds of it being slashed are quite low the odds of it going offline are quite low and that kind of gives you a more firm base to actually provide guarantees to extra services and then you know going a little further oh ball itself could reward and penalize people within a distributed validator by using

this kind of restaking and economic security and this is you know the kind of further version of you know in the near-term mobile can be a Staker for something like eigenlayer but in the longer term it can also be an additional validated service that's you know being the one paying for economic security or for restag to keep it you know protocol running um so yeah we're you know quite bullish on the idea of having distributed validators be a safe base to restake on yeah that's that's really cool I like personally was thinking you know also since you know I guess there could be a lot of additional services that um are offered through eigen layer right and maybe for a single node operator it

can be hard to run so many additional services so I guess do you think it's possible that like for example a oval cluster sort of shares these responsibilities like one guy runs the Oracle the other that's 100 right yeah in the in the near term there'd be a bit of trust involved if you're you know saying hey we will put our withdrawal contract you know towards eigenlayer we'll trust meher is going to run that Oracle for us he's not going to get a slashed you can do that absolutely and you know we could all be taking the risk of you know doing that extra additional voluntary service each um but going even further we're working on the cryptography to make kind of proofs of kind of you know Fair participation more easily provable and here's the the like the longer term goal

is that you know one piece of this distributed validator could opt into this service for everyone and if they do screw up they're the one that takes the the hit or it it's you know most the blame rather than the whole group sharing in in the like slashing if if you know one person who they like trusted to you know do some extra service for them doesn't like match but yes you you absolutely can have you know one person in your cluster doing all of these extra validating services for you yeah maybe this eigen layer ecosystem will be so large that specialization of Labor uh will emerge meaning some validators do some things and other evaluators do

other things and if that kind of specialization of Labor emerges then overall is kind of perfectly fit for for exploiting that specification specialization but it's still early days right like there's not not a single service that's actually running on this restaking uh model in Productions yeah I think they announced their no not in production they announced their spec and stuff I think this week so um shout out to them for that so yeah they're working away at it we're keeping a knife chat away is is the overall concept only applicable to ethereum because of its special staking model or is it also applicable to other chains

could you expand to other chains and deliver utility yeah it's a good question so look we've been thinking about DVT for ethereum validators for three and a half years at this point so um I made it a pretty large effort at the end of last year to be like hey guys it's time uh we have to go look at where else we can go right like this technology most likely will benefit other ecosystems or other layers of ethereum for example so we went on this effort to kind of go find problems recently so we spent the entire fourth quarter um and a good portion of the first quarter on a cosmos effort um which you know the team at course one

was very helpful with and a lot of the other Cosmos ecosystem as well we've also over time when we did like our first and second fundraising round kind of had this emphasis on like what is a second ecosystem that we think they're smart people working in that have the right you know Mission and like the right vision and that was Cosmos investors so uh the ecosystem itself already has a pretty actually large Cosmos presence in it uh today as um as opal um so we decided to double down on that fact because we had the best access to information and and we figured we could like learn as quickly as possible succeed quickly fail quickly kind of one of those you know mindsets um so we went into Cosmos and we started looking for problems um we spoke to the validators we uh

spoke to the liquid staking projects we spoke to the ICF we spoke to the builders program we kind of did this whole Loop of going through the whole Cosmos ecosystem and speaking to people and looking for problems um and we were able to identify you know a couple structural problems um as to how uh DBT could be useful for smaller validators for example in Cosmos to team up to get more delegation to enter the active Set uh there's kind of like this larger portion of Cosmos validators that are like chronically unprofitable even like some of the largest uh you know Cosmos validators aren't necessarily that profitable uh today either so the smaller to you know middle tier ones there's really no game for them unless they could maybe team up together right and attract delegation

through you know means of using T so yeah we found some good problems in Cosmos we published a blog post on it recently um you know there's coordination difficulties to making it a reality there's protocol difficulties there's cryptography differences for example Cosmos does not use BLS signatures but we also learned through that effort that if you wanted to get it done you could and there's kind of like a existing versions in the cosmos ecosystems that are about you know a third of what DBT is uh but there would be a lot of builds uh inside of Cosmos that would be necessary one of the most interesting things that we found about Cosmos was actually the kind of social threshold of the value of DVT and inside of Cosmos

what we realized is that there wasn't enough value at risk for people to think that it was something that shouldn't just be you know an open source primitive they couldn't believe that it needed to be a network with all these you know tools and education and funding and ecosystem around it um but our take from that was is that the cosmos ecosystem is almost mature enough to have enough value at risk for everyone to say like hey we think this is a good technology to adopt into our community um so for us we we believe that it's a little bit early for DBT in Cosmos but um we are still actively like working on

implementation-based research now so we like stated the problems and now what we want to present to everyone it's kind of like here's how it could be implemented um so yeah that's like one area of alternative layer ones outside of ethereum where our our communities aligned and we're experimenting to see if the cothos Community Values it and then we'll you know collaborate with it um the other area which is actually super exciting uh figment just put out a piece yesterday around distributed sequencer technology a new term that we're trying to coin uh and this has been the other research area for us at opal is just going up the stack and looking at your other actors right so today we've spent all this time on the validator well let's go up to the block

Builder let's see if there's like centralization problems around block building people talk about NPC block building right all these different constructs and then we've also been looking at the sequencer and like the prover the verifier all of these new actors that are becoming more and more important in the core ethereum infrastructure stack and looking at their adoption cycles and saying like is there a need or is there a reuse case for the work that we've done for distributed validators and all the stuff that we know we've learned from an external project building it not not from a foundation perspective actually which is the reality of where Layer Two is they're a bit more mature in there we sit outside the foundation Nation we do our own things and we do whatever um so you know collaborating with those

new groups of people uh has been quite an interesting experience for us so yeah we're we're actively looking at two different growth strategies for DVT to see if they can be helpful elsewhere uh one of them is horizontal uh that's Cosmos one of them is vertical uh that's going into the sequencer topic and seeing if our technology can be helpful there yeah very interesting I guess I would also like just like to hear your thoughts I guess you're like very close in the end to like both the validator ecosystem and sort of the other protocols and and I would just like sort of like to hear how you currently think of the the validator ecosystem and how do you see it evolving I mean obviously

sort of like one of the core ideas of ethereum is that there are these homesteakers like do you do you think that like there are much of those and will this grow obviously or will potentially might help uh increase that but I guess just generally what's your view of the space yeah um I've been surprised thoroughly at the number of like hybrid at home validators I I would call them right they're basically like small groups of at-home validators who are starting small companies together uh and there are tens and twenties and they're like growing by the number right we kind of call them like the tail end uh

validator segment uh and that's probably growing the most right we're seeing a variety of new and small valuator entities pop up um we're starting to see people from other ecosystems come to ethereum and by means of coming to ethereum they're doing so through DBT as actually their first knowledge base uh which was also like a very interesting thing and so now we're onboarding validators from other ecosystems not into the core configuration but into the dvt-based configuration uh from from the very beginning um but are probably some of the most like bullish honestly and exciting conversations we have with people are in the like hey me and my three friends just got together we started a company we're looking to like you know get involved as a validator and Lido we're looking to try to qualify uh there's

there's a variety of small and mid-tier validators that are trying to get voted into Lido and by means of increasing their chances they're spending a lot of time with Obel and dvts and to make sure that they're proficient and that make sure that they're educated by it so honestly there's a lot of really good inertia at the smaller end of the validator of embracing DDT and like using it to advance themselves so they can become more professional and this has surprised me uh this is not you know uh like a like a segment that I thought would be such a core user and like Pusher of the protocol but I think it's really like a it's a sign of like the ethos of ethereum it's it's like kind of

crazy to see actually right like that group of people feels empowered to go build companies so that's what they're doing and then they're using opal as a further like empowering mechanism to achieve those goals uh and it's quite fascinating so in the ethereum roadmap we have this uh this dang sharding concept that's coming in where essentially these validators will end up becoming responsible for data availability so if you have like a single centralized validator it's kind of probably easy to understand we have to store some some kind of data on our side but how does it interact with an overall cluster where there

might be three like three different validators does it data now need to be replicated across all three good uh really in each technical question mayor um so on this tank sharing is coming out in two phases the first one being Proto tank sharding and the second one being full dank sharding um in Frodo dank sharding they don't go to the extreme where they don't have every node in the blockchain kind of keeping and gossiping like all the data availability it's still you know everyone sends everything everywhere and If eventually with you know full dank sharding you're signing data availability witnesses to prove that yes

I know I did see this data it was available um oh yeah over the longer run um there starts to be you know trust assumptions between the the four nodes trust them strong word but they have to decide did we all see this or did at least any of us see this bit of data blob so that we can include it in our blog and like like sign to it but in the near term in protodank starting when it first comes in it's more or less the best effort for data availability it's done at the consensus layer and it's not you know you won't be slashed forward to my understanding or at least they're not putting they're not going to the scenario where you know all nodes don't store everything which is the first way it comes in so yeah when it comes into prototank sharding not currently a

problem when it goes to fulltank sharding the nodes do have to say have we really seen this and you know we have to kind of prove you saw it or as you said if you don't trust one another everyone see the data availability before we sign off if you want to be more cautious rather than more trusting but yeah it's um in in the near term it should be a problem and then also with proposal Builder separation a lot of the complexities of dank sharding is on the Builder and the proposals it kind of keeps the proposals relatively simple they just have to kind of propose the really fancy block that a builder made for them maybe I'll I'll end with kind of one of the conceptual doubts I have with this entire ethereum L2 roadmap

which is that sometimes I feel that e v meaning like you need the validators and like you all of these middleware Builders we spent so much time and effort building various products to make the ethereum validation system work and be decentralized so many thought Cycles have been spent here and now when it comes to the question of scaling ethereum most of our effort is kind of it feels useless because it's going to be scaled through the l2s now these l2s are going to be running sequences and that's like a completely different kind of

validation stack that's that's being built and like now you have now for example overall instead of reusing your work for ethereum directly to scale ethereum in in L2 you now have to go and think of this new concept of a distributor sequence are you wrongly need to write new software same for us Etc uh do you in some sense think your uh are kind of like not being wasted but by being underutilized by the ethereum ecosystem yet throwing all my favorite questions my hair um because I actually would suggest that we don't have to throw stuff away and

you know build new software and stuff and this um is the idea of ethereum equivalence that I think was kind of first coined Again by optimism but they kind of astutely realize that if they stay as close to the ethereum execution architecture as possible it's easier for you know to gain adoption and network effects and be able to reuse all of the existing L1 stack um for example their very first kind of MVP will call it was you know a modified get um not unlike um navigate to some extent and then recently or just in the coming weeks they're moving over to optimism bedrock and the main difference there is they've abstracted all of their kind of

code for the optimism kind of fraud proof game um into what they're kind of calling a consensus client and they talk only through the engine API the standard one that all of the execution clients use and adopting this kind of standardized API for them has allowed them to go from having you know op get to also having op Aragon and ultimately probably all of the existing L1 execution clients in the optimism world and we recently like announced a Blog with figment just yesterday on distributed sequencers and the Crux of the argument is exactly um l2s have already kind of competed on ethereum equivalence at the execution layer the next step in our head is to

add an ethereum equivalent Beacon API to their code bases so adopt PLS signatures adopt that API dumb it down a little it doesn't need attestations because it's not a proof of stake game but just have you know proposals using that standardized API and the benefit of that is you do get to reuse more or less everything from the L1 particularly from the L1 staking side the ones probably most important is the private key manage inside of things so you know private keys are always the most important thing so if you if let's say optimism for Simplicity they add a beacon API to their L2 like op node stack they can reuse web3 signer they can reuse jerk

and you know the benefit for oval if they can reuse distributed validators because it uses the same API and they can more or less use it all out of the box so yeah I hear where you're coming from being like should L2 is really invent the wheel and I agree with you they probably shouldn't they should you know copy the L1 apis as much as possible and then they get to kind of tap into the network effects of code that's already built and already been used so yeah our pitch to them is you don't have to do all of the decentralizing sequencer work yourselves you can keep it a little simpler have it be kind of a ram Robin thing with BLS signatures but then use distributed validators to actually go from you know 10 entities in round robin to actually 10 distributed sequencers in round robin

um type of setup so yeah I I degree that you know L2 I think you know generally will adopt this they kind of see the network effects trying to build something new and convince everyone to come use your stuff as hard trying to you know stay aligned with existing apis more work for them but you know more Network effects and easier adoption for their customers and for everyone else right so actually actually this then seems very powerful for the overall Network that you can effectively go to an L2 that has let's say a centralized sequencer and maybe your product is simply that instead of now this being a centralized sequencer it's actually distributed across five parties or six parties and then if they can work out how to go from

one centralized sequencer to five and each of them now is an o-ball validator with like five or seven then you can Autumn easily get to 25 or 35 people running the infrastructure exactly right so so all of your work done for ethereum kind of um yeah scales like probably is more useful on the L2 layer then maybe or even on the ethereum main net like that could be a future for of that could happen for the overall project yeah that was actually something I also pitched where seek like in the L1 is normally very small machines and there's technically you know 500 000 validators

but sequencers they often will be running like very large machines so you know five very large machines running a distributed sequencer setup makes a lot of sense rather than you know thousands of small machines the l2s mostly won't optimize for like running on consumer Hardware they're you know meant to be a scaling solution but they do want to be you know as trust minimized as possible so yeah big sequencers that are run in a distributed validator type of setup where if one of them becomes malicious nothing happens they just you know like stay online like a multi-thig is it yeah I actually agree I think it nearly makes more sense at L2 than L1 um by the majority side of things perfect

and yeah awesome that we went very deep thanks so much for a dank episode Oceanic Colin and yeah thanks for for coming on the epicenter maybe yeah before you wrap up uh you wanna shout out uh where people can learn more about obol or or yeah how they can find you and yeah again thanks so much for coming on yeah thank you guys for having us it's it's been a great discussion um yeah to find out more about opal best place to probably link into the communities through Twitter you can just find us at opal Network and uh we have a cool ambassador program we have a great Discord we just well tomorrow we're actually monitoring our Research Forum uh with the distributed sequencer piece as the first one for people to get some

debate on and then we'll actually build out the cosmos section of the Research Forum so I think you guys are interested in getting involved there um in the next well this is well yeah in the coming days you'll be able to see it um and yeah I'll pitch doshin but thank you guys for having us and really appreciate it yeah um nothing major to add thank you very much this has been super fun to talk in the nitty-gritty of validators because normally we don't get to talk about you know bare metal versus Cloud versus Cosmos or L2 and their architectures so I'm gonna be grateful of this chance for to get really into the technical nitty-gritty in podcasts I hope they're all as technical as this one