The challenges of building, scaling and securing DeFi protocols

Transcript

[Applause] we have Hari from spear bit moderating the panel Ernesto from a Sarah Reynolds from uniswap Eric from Oiler and Merlin from Moro hello can you hear me is that all all good okay perfect hello so let's get started right away um So Def as a concept was invented about six years ago and since then it's quite mature today and I wanted to bring some of the largest def5 protocols out here and ask about their challenges in building

scaling and securing their protocols so today I have two protocols that are you know con synonymous to the concept of defi a and unisa and then I have two more who are you know younger hungrier and like amb protocols uh Moro and Oiler I wanted to learn about their challenges in building the protocol and uh what we can all learn from it so let me start with U um anesto um I genuinely think he has one of the hardest jobs in crypto maintaining the a protocol but we will get to that in a moment uh but before that you know I would like to I would like uh for you to tell everyone in the

in the audience you know what you do at a and what is is your background yeah so yeah uh I'm Ernesto as har say like I'm yeah the co-founder of of a company called BD labs and we are one of the contributors to a like do core development maintainance and so on of the protocol and well I mean my my story goes from back in the days like of the six years that that har mentioned because yeah I was part of ab a back then in 2018 when AB I think it was not called even ab and and yeah basically what I do is so is everything related with tech on

AB so that includes major like what surrounds the the smart contracts which can be upgrades of the protocol like the sign of certain new features but also basically everything around like so more related with this conference like um coordinating the security ER so thinking uh what new tools to apply what new procedures to apply and and so on and so forth so yeah to summarize I I do a bit of everything related with the protocol okay great so moving on to Merin um you ended up building one of the hottest projects of this year which is uh Moro

uh tell us your story and how you ended up building Moro right um so I think if I remember I I met Paul during if Global aaton I think it was like during the the confinement and um he had this uh later on he reached to me and um he wanted to build like a new protocol um a sec a researcher at the time had the first idea of Optimizer uh M Optimizer which was basically like building and uh on top of existing Landing pools and try try to improve rates for both suppliers and bard so uh Paul wanted to work on that idea and uh so he reached out to me at this

moment I was uh building and trying some stuff in the def space I was uh playing with the credit delegation on a uh it was really fun and I wanted to create a project in in that space so he called me and I was like okay let's go uh St align so um and so we created the the first version like after maybe one year after during our um our studies and uh so we launched I think first Moro compound which was optimizing rates on top of compound and uh after we uh we we launch on on top of a and so that that started and I think like during bur Market uh we reached like two billions of deposited ass set on on

optimizers okay great moving on to Eric he has an interesting background coming from VAP to security and he joined Oiler after the unfortunate hack you know tell us your story and how you started working at Oiler um I guess like uh so I'm the head of cyber security at Oiler Labs um I I was in the web3 gaming industry working as the head of cyber security there there were kind of the market wasn't doing well so they were doing layoffs and I came up with the opportunity to go work at defi which I was super excited cuz I love trading investing and uh I thought it was a great opportunity

and uh yeah so I started working at Oiler uh actually found a bug my on my interview they were doing um oh I don't know if I can talk about it but uh they were um not validating some input and I quickly found that I could just like spam it and uh I got the job um and uh yeah so I manage the organizational security I do lot of the stuff that paito e was talking about like uh ensuring that we're running antivirus and all the good stuff that he mentioned uh and uh I organized all the security audits the pen testing um and

yeah it's been a great experience working there okay moving on to Sarah I think you have been quite involved lately on pushing transion storage to you know everyone's uh you know I know I know you were pushing it in the solidity compiler and convincing other people to use it as well tell us your background and the work you do at unisab yes hi I'm Sarah I'm one of the senior protocol engineers at Unis swap um most recently I've been working on V4 and even before V4 like Hari said I was working on really trying to Rally the community around transient storage I think it unlocked a lot for um

developers like myself um yeah before un swap I was in college so great that's my story so maybe starting with uh you know you as well so what is something that you believe the unisa team does that um nobody else is doing out there and something that you believe is like truly unique and I would ask the same question to everyone but replacing unisoft with you know your own team so tell us something that you do that's unique to your team and you think everyone else should be doing that's a hard one off the bat um I think that we it's hard to pick out a specific unique thing that we do I can maybe share like some unique stories

um but I think you know we're all in the similar boat here of kind of building protocols in the open um we have a lot of eyes on our protocols all the time so I think there's a lot of I think similarities we share um one I guess unique story um as we were going through the audit process um there were moments where a lot of the engineers on the team you know we had fixes for some of these things but they weren't great we didn't feel really good about them and you know in that case what do you do I think we were going to have to like settle with um kind of a fix and one thing that's really cool I think that all of us shares we we're

building these things in the open we can have conversations in the open and we really leveraged that so I think we the this was at eth Prague maybe I'm forgetting which conference but one of a Community member basically proposed a solution to one of our critical um issues and that's actually the current solution in the code today and so I think like really leaning into kind of building in the open and um leveraging like sort of diverse expertise in a lot of different areas is something that I our team does and has worked well for us yeah I think that's a great story like I see there's tons of

PRS in the unisab repo all the time from random developers some of them gets in some of them doesn't but it's always good to have uh that kind of community getting engaged into building a protocol what about you Eric what is something that Oiler uniquely does I think um I mean we do many things but I think we start like internal reviews I think uh that's pretty unique I don't I bet you guys also do it so I don't know uh we start with internal reviews we start like uh even before we finalized the code we started like early we started like a VC so review even the code wasn't finished but we started getting feedback uh on like the

architecture the security design um something I think that is very unique is that we did a live CTF which is it was very difficult um and uh high risk but we we decided to put our money in first to kind of like actually try it out and see if anybody could take it um I think that was you know yeah I mean I know the behind the scenes behind would you mind explaining to everyone what you did I think that's a fascinating story uh yeah so like uh in order to like Kickstart Oiler um we decided that it like it would be a good idea to actually put our funds from the government to start the first landing pools and uh we wanted to be like super safe cross all

the te like three times we already went through a a Cantina competition we were still confident but we wanted to be like super 100% sure so we did a live CTF uh with hats and the idea was to put uh 5 million in and if anybody could take it they could take it uh and the idea was like you know try your best to break it uh and uh it was an exciting and scary moment um also it it would it's difficult with regulations and like trying to figure out like how to make it um in a way that you know we're putting money out there and like protecting it and so we had a very diligent process

like reviewing the con I mean the terms of service of how this would operate which I think it's been one of the toughest things but it was a thrill and I'm glad that like we didn't lose anything yeah I mean that's good to know I I always always find it hard when in Engineers want to do stuff but the legal comes and say you can't do this it's very annoying um so so on moving on to you Marin what is something that's you know unique to Mor for right um it's good question so we we do internal reviews but we haven't done the the the CTF um which is like a good thing to to try maybe next time uh anyway so I I think like um one thing that is

different from on M for it's so we we've went to from building optimizers on top of a to like creating our um own Landing uh protocol and I think like uh the the way we uh view how to build like a safe and scalable protocols have evolved a lot uh during those three two three years and now I I I'd say that the approach of mofu of building protocols is um like really about like minimalism um trying to do just one thing but do it perfect perfectly uh trying to remove as much as possible uh the governance in the in the process in fact it's really similar to how Unis swap like is

building the the protocol since the beginning like simple code um uh trying to externalize like the the thing that you don't want to see in your protocol like the or oracles for uh dexes for Lending protocol can be like a stable rates or leverage features Etc all those kind of things can be externalized and in fact you just like refine and try to simplify as much as possible your protocol so that at even before starting the the writing the lines of code you have just all maybe removed like 90% uh of the the the likelihood and the attack surface uh and I think this is like a very good approach to um make sure that

you starting like the right way and and uh make like all the process of like stacking security layers after much much easier what about you anesto what is something that's unique to a I mean it's one of the most battle tested protocols out there I I would say like like the the thing that probably differentiates a is that we know how to find equilibrium which is a bit abstract but what I mean is that when you are like for quite some time like in production and very high state Tak is it's pretty important to understand like how far we are with new features you go

with new features versus how much you are conservative or how much pressure you receive from competitors like from the market versus how much you need to take care of let say let's say your back office of clean up and so on so I think that that is one of the of the keys of the success of a because we know pretty well like on when we should improve something or when we feel that maybe that should not be touch and and still keeping the Rhythm like with with everybody else in in in the market there are then probably some other aspects that I think a is pretty

different but that is maybe some that is not not so so told publicly so frequently that makes sense so I have a followup question to that which is uh so people talk about this idea of Defense in depth and security it's the idea that you do multiple things hoping that even if you miss somewhere it will be caught at some layer and there was a great story about how a did this um and the story was about how a discovered a bug in the llvm compiler framework so for those who don't know llvm is a compiler framework for creating new programming languages and for some reason I Ed up up finding a bug in the llvm then it had no

place in like finding a bug there so I want to I want to learn about the story from anos so how did you find a bug in how did your team find a bug in llbm yeah I mean first like I maybe will summarize a bit how how was The Bu so basically what happened is that AB as you probably know like is in multiple networks and we we keep expanding like the V3 protocol to to new networks and so this was in the context of expanding to CK sync and well the process is the contracts get deployed the Dow needs to activate and be a governance proposal so on and so forth but what we tried to achieve always is that the Comm the code is kind of the same like or or completely the same across all the networks so this case was very strange because we there

was a deployment on CKC there was an activation everything seemed to be working perfectly fine and then in the yeah first seconds or so um it was not notice like I will elaborate on that just after like it was noticed that very core a very core mechanis of ab was not working well like the related with the collateralization which is basically the most critical aspect of of the protocol so H well it happened to be that after paing the protocol and so on that was a bug in the compilation stage of the uh llbm which is like uh the back end library for the compiler used by the CK sync compiler so very very deep thing um so yeah like when you tell that story it

looks that yeah other guys are are so smart like they they found like that but I would like to highlight this is quite different what happened like I think that the key on why the The Bu was found and this is related with well this ethos of ethereum and of the community of saying yeah the centralization and so on and so forth this B was probably found because of the decentralization of ab because um it's not only one team like in practice like who is involved into the activation process of a new a pool like is okay we do the deployment on BD Labs we do a lot of testing and so on on the initial stage but then we have likeor which does independent reviews we have Abel apps

which for example in this particular case what happened was that all the test were fine like the formal verification let's say was not possible to completely apply but seemed fine like from the cura side the review and then while testing like the user interface like one particular Edge case like in this case have aabs that maintains the interface they notice like that edge case and then that initiated like the whole thing like of okay what is happening here but first we we pause then we try to understand but I think that that was the strong point because there are certain types of BU that you it doesn't matter how smart you are like or even how much procedures a single company can have like like when you have

multiple entities involved into the chain then is when when those Pro problems are detected so I think that is like a yeah a good point of view for yeah I think that's a fascinating story for so llbm framework is used by like pretty much every big compiler like clang like or or like pretty much every programming language out there used to it um in in the vapu space so very fascinating to see like a building solidity ended up finding a bucket llbm and the story is you know always fascinating um so going on to Moro so so at speit we have worked with moru for a very long time we met them back when they were a very young protocol that had a simple idea let's optimize the rates

uh for Lending protocols um and um since then Moro evolved quite a bit you know they went from working on top of compound and a to having an entirely different philosophy on how lending Protocols are designed so how did your team change your engineering culture uh during this you know massive shift all right um so first we so we deploy the the optimizer on top of L uh compound and and a and in fact we realized like it we were like super deep into the the code base of those both protocols and like very at the same

time like providing a landing experience for our of our users and I think we are also the user like as a protocol like the closest to uh the underlying protocol uh a or compound so we had like those two two different views um as a both a user and uh a landing protocol and we realized that it was kind of hard to build on top of a upgradable smart contract uh when you were so deep because like or like a contract that can be updated quite frequently because like any change to reeters or um to of the code base of the logic like can impact you and like even

break your um uh break your integration or like a block some features for your users Etc so we had to have like a constant monitoring of of that whenever there was like a posing some something that we we needed to jump in wall and make sure that everything is smooth um and it it proved to be like super challenging for us uh and also we were like um facing directly like what are the limit of like the current design and like a current Paradigm of building Landing protocols and so at some point we we we told ourselves like hey instead instead of trying to improve like what's uh what is the the current Optimizer stack like that just build our a landing

protocol like the one that we would have liked to integrate and let's define like what are the the design principles and like what what really we want uh this primitive to do and that that uh that's uh how we end up like ended up like a building our own lending protocol that is called muffel Blue that we've deployed like early um February this year I don't remember exactly uh but U which is like totally immutable permissionless um uh governance minimized where Market are isolated etc etc but that's that's and how did your teams like engineering culture change you know you you went from building this

upgradable contracts to fully mutable so I think like uh we we started like without bias uh so we we don't have a finance uh background like uh most of the founding team like we haven't worked for another company uh and except our when um so um Paul and and I like we started like on both side like trying to to launch like startups Etc um so I think we had no bias so we were not like um we we we didn't have like any Affinity to like upgradable contract or immutable contract or a specific design and so we just like uh welcome like criticism and tried always to ask

ourself like hey is there like a better way to to to do like a landing and and if there there's like the answer is yes like so let's just like dive deep and try to find like how to to make it better instead of like trying to uh to stick to like uh one one way of thinking that makes sense um moving on to Eric so the history of the oiler protocol is a bit of a roller coaster there were a truly Innovative protocol uh that innovator in lending back when everyone was trying to launch an A or a compound fork and Oiler was like one of

the few protocols that truly innovated in what they were building and they they had a great team they put a lot of work into security but they had a very unfortunate incident um you know the oiler hack and the roller coaster was and they managed to get all the money back in fact I think at some point the money was even bigger than um what was lost so tell us the story about uh the oiler hack and you know some of the recovery and how you came into the team and moving on from that like you ended up building a version two of the oiler protocol which I think is one of the most hardened protocols in crypto today considering all the work they put into security you know including the $5

million onchain CDF where anyone can try to hack and keep the money if they do it so tell us a story of how the oiler went from going through the hack to recovering it to building one of the hard hot most hotend protocols out there yeah um I mean I wasn't part of the company uh so I mean what I can say is that uh like it was probably very devastating to my teammates um it was difficult it was definitely roller coaster uh I think if you really want to learn more about it I recommend reading the very well detailed called war in peace it's a block post by Michael really detail about like his experience

it also has personal like side to it like I mean I think his he just has a newborn and all this things was happening and the negotiation and everything that went part of it um but luckily yeah we were able to recover all the funds and try to make right with everybody um after that um I guess like the team decided that they wanted to continue um and uh they went into building initially we just wanted to fix the the the bug and relaunch but then like I guess they got inspired about how to make it better make it more flexible make it more modular um we wanted to they wanted to build like something

completely new and creative and be more robust and so we went on this giant Journey that then I became part of um I think uh I never seen I mean when I joined I was like pretty impressed of like the commitment we threw like three kitchen sink or even like 10 time kitchen s and to security uh the other day I was doing the math we did 700 days of audits oh wow like it's more time than I've been in the company if you add it up uh we've done like amazing work working with spearit chain security Sor formal investigation for like six months it was great help like creating all this

invariants and all the work that we yoted has also been an excellent partner um and uh it it was really cool because like so as things were getting finished we started audits like for example DC got audited about like nine times then once we had the EVC which was one of like the core components then we started working in the evk then we started audits with spear bit chain security y audit then there were smaller components so we work with other companies like Omnia then we kept on building uh then we wor started working on the priz foral and so like all the code got like at least least I think our philosophy was not to just rely on one single auditor but like try to get as many um I into the the the development and design and

then like then we went part of like doing the code competition with Cantina which was a another roller coaster cuz we wanted to launch soon but we got like uh I guess 472 issues uh 500 plus participants and uh we went through each uh reports took our time commented it was really fun working with the community uh it was stressful for my team try because we put like uh a lot of time into our respes um and then uh yeah we followed by the live CTF and now we're working on like onchain monitoring and just making sure that everything is working perfectly we

have uh some cool like features like if we ever detect like a dpeg or any asset deegs we can like set Supply caps to zero so we can unwind markets and there's a lot of things we're doing after the fact for like Ina response and um I'm like you know really proud of what we end up building like it's pretty amazing also the internal security that we're managing we're doing Web Two pan tests and like um I could keep going on and on but yeah it's been a roller coaster definitely Harry yeah I mean I I I thank you for being part of it yeah I can totally imagine I feel like Oiler V2 at this point is one of the Harden you know protocols I I can't imagine any any

other protocol putting in more work uh to security um that brings me to unisab um so you know unap is pretty much synonymous to defi today and back in 2022 um there was this ethereum merge and and the whole idea of the merge was you switch out the consensus protocol live um and it you know there was like few months ago or maybe like few weeks ago there was the the rocket launch uh of the SpaceX and it came back to Earth and somebody was asking the question what is the software equivalent of a rocket going up and launching back um intact and somebody said it's the ethereum merge and I was thinking about

how every few years UNICEF has to go through something similar where they have a perfectly working protocol trillion trillions of dollars in volume and they have to suddenly switch out to an entirely new uh protocol uh expecting billions of dollars in uh tvl in like you know a couple of days so how does the unisoft team you know do this swits every few years yeah it's a really good question and I really like that analogy because I think this is similar for unop but other protocols as well of our development process is definitely more akin to building Hardware it's a release and there's no upgrades after that and so I think that's also a feature it's an

opt-in upgrade for users and that's how we like to frame it as well and so as we're going into the launch of V4 that's kind of our framing is you know users don't have to use V4 if there are compelling reasons too you should um and you know we definitely think that there are um but yeah it's it's definitely I think sort of the nature of building really a mutable protocols and how do how do you how does a team handle all the stress of like you know you launch it and you expect in couple of days that a billion dollars are going to flow into this protocol like like what does your team do that I feel like very few teams in in crypto

has to manage that level of uh Insanity yeah um I think you know listening to Oilers kind of process on like hardening I think we definitely didn't spend 700 days um but we did put in a lot of time I think back in March we had like a um a whole like fiveish month plan outlined for what all the audit processes look like um we're going to start monitoring I mean there's it's it's everything I think everybody here has done um and it is a lot of stress but at the end of the day I think it doesn't lie on one person it's like a lot lot of different people

contributing to this and it goes through a lot of different eyes um and so we have to have trust in our ability yeah amazing so I want to give some time to the audience to ask one or two questions so if anyone has a question to like one of the team members this is your time cheers yeah uh this this is for Sarah I was just wondering with V4 um like when you're building out hooks what are like some of the common uh issues you see with people building hooks or security issues that you see from like

yeah people just getting into hooks in the beginning yeah that's a great question um I haven't done any specific like audits around any particular hooks um I think what I guess off the top of my head what I would talk about is certain checks that you should be doing in hooks like a really simple easy one is like checking that the caller is the pool manager that's a really simple check like every hook should have but if you're new to writing hooks maybe you wouldn't know um another thing to think about I think is like the interaction of different hook permissions um one thing that I'll talk about later today is how I think one principle in building hooks is limiting

those permissions and that permission set so very naive hook developer might you know okay let's you know make an upgradeable hook and enable all the permissions and we're going to change you know um you know what when the pool is calling out to my hook that's definitely a scary hook and I think for a lot of users and a lot of other routers and other integration contracts down the line is going to be something really hard to integrate with um and so lean into I guess I think all that is to say is lean into the side of building hooks where um you have predetermined kind of permissions and flags and things that you can do in your

hook and if you ever need something you know different in that hook it's really easy to do a new pool and um I imagine seeing a a a V4 where there's lots of different pools lots of different hooks but it's sort of per pool um yeah that's what I would say yeah and that makes sense and and just one followup I guess uh how do you guys envisage like uh how people will access hooks once V4 goes live like will you guys have sort of a directory of hooks that you think are safe or like how do you sort of see like how you'll direct people to hooks that you know should be okay to interact with yeah so I think about hooks a lot like tokens um so we built token lists and I

think that something similar could happen for hooks right now we're working with the foundation on kind of a establishing um sort of a standard around like hooks interfaces and this will be really great so front ends can build out like hey here's a hooks directory of like how to learn about Hooks and how to verify it actually on Chain by like calling into a contract um that's still being worked on but I could imagine it looking something like a IH hooks metadata interface or something like that perfect thank you questions there oh great panel thank you so this sort of a flip sided question right majority of the security conference is

really about you know us security researchers preaching talking about how protocols should do better developers should do better users should do better but on the flip side what do you protocol teams think that we should do better as in the security community in general maybe the security researchers or security projects what can we do better to support you know your Dev life cycle make your uh lives less chaotic less intense yeah maybe I maybe I can comment yeah I think you know I I think that sometimes there is too much disconnection between like the the security community and and sometimes it

even feels that he growing as as more security like experts enter into the field there is more disconnection between them and like the Developers of the protocol which is I would say quite unnatural especially in this field because like we as developers like are I almost there is no separation between development and security especially like in protocols like can it be AB because I mean obviously you you design the things and security considerations are maybe like yeah 60% of of of the whole design like so I think that this a good exercise like for security researchers to try to always put themselves like in the position of the developer like try to understand like because yeah maybe

the documentation sometimes is not so good like in some projects but but do a lot of questions to the developers like and try to understand like the point of view of why something was done the way it was done like because yeah I mean maybe with some teams that don't have much experience like it's just by default but knowing even that is very valuable but with with teams that have pretty good experience or like strong positions is good way to learn like of okay what what is the developer thinking like why maybe there was this mistake done that created this vulnerability so I think yeah like improving the communication probably is uh yeah something that that will give a lot of

value to security researchers okay uh what is the like one thing that each of you worry about at night like what's the what's the mo the highest risk for each of you on your protocols s sorry I I I didn't get sorry what keeps you up at night H what what keeps you at night you know do do you sleep well and you know if not like what is keeping you up a nice um sorry what is just the like one highest risk for all of you that that you worry about the most what keep us awake yeah that keeps you awake at night uh no

internet disconnected yeah I mean I don't know I think that is actually something very important like because is what har said like it's a bit insanity if you think too much like on what is at the stake like you will just go to a cave and not do anything anymore like so I think it's just very important to keep yeah the equilibrium that I mentioned before but also like personal equilibrium in in the sense that you need to be cool you need to understand that okay like some some problem can happen some some back can happen like but if if you know that you did things like right of what you believe is right usually I mean should be fine like I I personally don't have any any problem to sleep at night because of that like I know that there

is like high responsibility like but but I mean it's only negative if you give too too much importance right so same I think we we are out of time to okay so thanks everyone for the panel I think it was you know great answers and we all learned so much from all of you thank you