The Unreasonable Effectiveness of Audit Contests

Transcript

here uh we'll just go ahead with Jack Sherlock from uh sorry Jack Sanford from Sherlock who's going to talk about yeah he's going to talk about the unreasonable effectiveness of audit contests cool uh thanks everyone for coming thanks for showing up appreciate it I know insurance is not always the most uh exciting topic ever but I promise that it's a very important one and it's going to get crypto to mass adoption so we need you guys to be uh to be here I appreciate it um so some quick background on Sherlock before I get into the talk Sherlock is a smart contract insurer um totally trustless on-chain insurer right now we are ensuring

um about 35 million dollars on chain for protocols such as open Euler um teller Tempest uh who else hook uh liquefy bunch of guys a bunch of newly launched protocols um so really excited about that and for Sherlock and for defy Insurance what we really really care about is making sure that Protocols are safe obviously and we don't have payouts and so that that's why we've gotten super deep into how do we prevent payouts how do we make sure code is safe and you know that mostly looks like audits right now although muli and sertora are doing formal

verification there's a bunch of other really exciting things that are audit adjacent that really help with security as well and one of them is audit contests and basically audit contests if you don't know what they are oops where's the uh if you don't know what they are maybe you've heard of code Arena that's probably the biggest audit contest provider or runner in the space right now uh they've been around for about two years or so year and a half and the way it works is they basically just say here's an eighty thousand dollar pool of money and

anybody who finds a bug in these certain smart contracts gets some of that money um so it's very it's sort of competitive it's open to anybody in the world to compete in these contests and based on the severity of the bugs that they find in those contracts they get paid some of that eighty thousand dollars um so this is a contest that Sherlock ran for our protocol I think back in January was an 80 000 pool here are the uh the top performers who got paid uh you know eleven thousand dollars and and um amounts in that range so that's basically what an audit contest is and Sherlock cares about this new type of

audit because we're always trying to figure out what is the safest way to secure protocols on chain what's the best way to secure protocols on chain so I went through every single audit contest that's ever been done for you guys and looked at the data to see how audit contests have performed um you know not necessarily compared to Audits and specific audit companies but just Standalone how have audit contests been doing and these slides will be available afterwards so you can go to that link and you can see this uh data set oops Okay so the first contest that was run by code

Arena was one and a half years ago they've run 154 unique contests so you can think of that as audits they've done 154 audits in that time they've done it for 110 different protocols as big as Ave open C and you know as small as anyone who wants to put up a twenty thousand dollar thirty thousand dollar pot so all across the Spectrum and on average um the contests have finished about seven and a half months ago that's just important because we're trying to figure out if these audit contests secured the code and so on average that code has been on chain for you can probably say six or seven months so that's a decent

amount of time for it to have had the possibility of being hacked so I think this is you know a fairly decent data set a lot of these protocols have been on chain long enough to see if the code is at least somewhat secure um so how many hacks do you guys think have happened in those 154 contests anybody three okay seven 42.

0 how many two thousand I like it um yeah so let me remember the number off my head but off the top of my head I believe it's 14 have been hacked since the contest but a lot of those were either front end hacks or admin key issues or things that were out of scope for code Arena so I went through every single contest I went through every single one of those hacks looked to see if they were basically code Arena's fault or not or within scope of the audit to see how code Arena actually performed how Auto contests actually performed [Music] um

so their seventh contest ever was the launch of me bits um you got some of you guys have probably heard about it or potentially heard about the uh the bug that happened on the uh I believe it was the first day or the first week of the me bits launch um basically the contract allowed people to Brute Force um rare me bits uh the me bits actually made it even easier by showing the data set of where the rare me bits would be um based on the sort of nft that you own I believe it was crypto punks and potentially one other that you could be bits with um and so this contract was audited by code arena in an audit contest but it

was not really code Arena's fault that this hack happened because they flagged it as a medium issue and they said hey random index is not truly random there's a possibility of minting a specific token ID basically there's a possibility that you could brute force uh the rare me bits and that's exactly what happened um and me bits for whatever reason decided not to fix this medium vulnerability so another one contest number nine visor Finance their code Arena contest finished on May 19th they were hacked three times in that year after that contest um June 19th and then in November and

December first one was an admin key was stolen so that's not really in scope of an audit the next one was the deposit basically used a spot price instead of using a t-wop something like that so again this was back last year when t-wops were a little bit less well-known or prevalent and then the third one was just a logic issue where the deposit contract or the deposit function asked the attackers contract to approve the attacker's contract um and obviously the attacker will approve its own contract so it can execute on the attack [Music] all three oops keep doing that all three of these

um hacks or issues were out of scope so it was not necessarily the fault of the audit contest itself it was not missed in the audit contest um and so you know those were both in the first 10 contests there were probably 10 people that competed in those nowadays we're seeing over 100 people compete in audit contests um but basically to answer the question of how many hacks have happened on that audited code in those audit contests um the answer is zero no hacks have happened in 154 audits with audit contests um and that I think that's pretty crazy

um the rest of Sherlock thinks that's pretty crazy too and it's pretty we think it's fairly statistically significant in terms of showing that audit contests can be a really valuable way to secure code when it goes to mainnet or before it goes to mainnet so audit contests are really great they have this really crazy raw horsepower where you can incentivize like a hundred people to come look at your code base but there are some issues with it uh basically you don't actually know that a talented auditor is going to show up on any given week and that can be a problem especially as more of these audit contests happen and some slip through the cracks or maybe

you don't incentivize the pool to be big enough for talented people to show up there's also no guarantee that let's say a talented person does show up and they show up in the rankings and you say great they spent time on the code base they could have spent an hour on the code base and maybe nobody you know it's like the bystander effect maybe everyone was busy or had a wedding or whatever that week and didn't actually look at your code base much so that's a that's a risk and the last one is there's no fixed reviews so basically when you do an audit usually there are some kind of issues and the protocol needs the protocol team needs to go and fix those issues um code Arena does not really offer that audit contest don't really offer that

sort of um functionality that that kind of fixed review and they often make the protocol team look at hundreds of issues a lot of them are duplicate issues um a lot of them are low informational issues things that you don't necessarily care about for securing the code on mainnet and it's it's a pretty big burden sometimes we've been through it with I think 400 issues and trying to go through figure out which ones are real which ones aren't it's not very easy so that's audit contests traditional Legacy audits kind of what we're all used to in this room um I'm sure you've heard about them it's two or three talented Auditors looking at a code base for a defined period of

time usually two weeks maybe four weeks something like that the problem with that approach is only two or three people look at your code and those two or three people are not perfect or they focus on certain vulnerabilities that are better at you know finding these things than those things and so there are often or sometimes holes when you only have two or three people look at the code base Sherlock experienced this ourselves we had a high severity bug we did a top tier audit from a top-tier audit firm and then we did a code Arena contest and the code Arena contest found a high severity bug in our code that could have caused a lot of problems for us on mainnet that was missed by the top sort of audit firm

um and the other crazy thing about that is the people who found that high severity bug were ranked number 30 and number 66 on code Arena so even if you've tried to get the top five guys on code we need to look at your code you still would have missed this bug um so you think that's kind of crazy some other problems with Legacy audits there's no direct incentive to secure the code base there's reputation everybody knows about that there's auditor reputation and it's kind of this amorphous thing that's been built over years um and one bug for one contest doesn't necessarily change an Auditor's reputation so there might not be that much incentive to do a really great job on your audit specifically and there's often long lead times but that can kind

of change its demand for audits fluctuates so at Sherlock we care about securing code as best as possible so that we do not have payouts on the insurance side we asked ourselves is there a way to get the best of both worlds where you can have top tier Auditors who are dedicated to your code base for a certain amount of time incentivized to find bugs but also have a contest pot where anybody in the world can show up and submit a vulnerability or find something crazy that only they know how to look for and get paid for it and some other things like can you have a fix review with it as well which audit contests don't really offer right now

can you have concisely reported issues which Auto contests don't really offer right now can you do on-demand scheduling which Legacy audits can be very difficult to do that whereas Auto contests you can start them you know within a couple of days from when you decide you want to do it um so basically Sherlock is launching this product um and it's going live tomorrow and it is totally transparent in terms of the pricing and the contest and how everything is going to work for a normal two we got it it's a 75k cost with a 50k contest pot so you can think about two-thirds of that audit cost goes to a contest pot and anybody in this room can come submit bugs and

get part of that 50K um then there's a top tier auditor which is basically the person that we are incentivizing to actually spend time on that code base for that amount of time and we pay them a fixed amount so that they will show up and we know that they're going to spend time much like a legacy audit and then the top tier auditor also does the fix review at the end so that's a really important thing you want to make sure that the fixes don't have issues themselves so you have to check that there's going to be an ELO style ranking system so to become a top tier auditor anybody in the world could do it anybody in this room could become a top tier auditor all you have to do is perform well in a couple of contests two to be precise and you can become a top tier

auditor and start making 20K every two weeks doing audits um and then of course Sherlock is an insurer so behind these audits because we feel so comfortable about the kind of security of this hybrid approach um we write 10 million of tvl coverage and we'll even cover your one million dollar bug Bounty through immunify after you get this audit so that's kind of the power of Sherlock that is the next what we think is the next generation of audits um and yeah basically the first contest launches tomorrow the sign ups just went live about an hour ago you can sign up at sherlock.xyz and again anybody in this room can join find a bug in the contract and get paid

thousands of dollars so definitely check it out go to the site see if you can find bugs and sentiment and uh there's going to be a lot more contests after that so that's it thanks guys hi um my question here oh what's the level of coordination between Auditors in a contest like with what I've been an auditor for some time and I know that lots of the value that you would get in Legacy audits is that the Auditors are working together right and they can exchange ideas

so I'm wondering in Sherlock me as an auditor would I be working in a team of Auditors or would I be working along thanks yeah that's a that's a really great question um basically there is a small trade-off there in terms of if you allow everybody to work together then it's very difficult to figure out who is contributing how much and pay somebody for finding a bug because your whole team found that bug and the way that Sherlock sort of handles that is you're allowed to sign up as a team or as an individual and so like with sentiment the the lead auditor on that I believe is going to be watch pug which

is a team a team of I believe three Auditors and so they work together they found that they're able to kind of collaborate best as a team instead of individually and maximize their earnings that way and so that is a 100 viable approach is to kind of find a team that you like work with somebody you like and decide okay we're going to compete as a team or we're going to compete as an individual I have a question about the insurance like how do you price your insurance yeah so there's been a lot of uh that's a great question there's been a lot of iterations on pricing with Sherlock with Nexus with a bunch of different insurers it's a very difficult thing the the root

of that problem is that there is not a good data set of payouts of claims of hacks even because D5 is like two years old whereas if you were trying to ensure you know hurricane risk in Florida you've got 100 Years of data and maybe you've adjusted slightly for global warming or something but you have a pretty good sense of what the you know worst damage looks like what the average damage looks like things like that we don't really have that in defy so it's much more difficult to know where that pricing should be and the way Sherlock does it is basically we say we have some audit requirements we have a checklist of audit requirements that a

protocol has to do before they get the audit from Sherlock and if you meet those requirements like having I think greater than 80 percent test coverage things like that then you're able to do the audit and as long as you can kind of get the fixes done within a half day period during the fix review then you know you go through that full process and everybody gets priced at two percent so we're just keeping it very simple for now in the future you know I think we very much will try to partner with people like Satora and lower that premium amount to one and a half or 1.75 for doing formal verification and a bunch of these other things um that also help secure a code base but

to keep it simple and to yeah basically keep it simple for protocol teams it's always two percent yeah I have a question here so um how do you determine when an instant happened how to determine whether that instant is due to let's say some external like sushi or whatever or is it due to the lack of inspection from the auditors um so the first part is how do you know if it's due to Sushi oh like uh Oracle whatever the external contract yeah yeah um how do we determine that it's it's fairly

um you know a security expert can go through and decide okay this was caused because of an oracle manipulation like the t-wop moved from uh whatever to you know infinity or the highest solidity number possible um so that's fairly determinable I assume your question is how do we decide it on the claim side like how if you're exactly yeah probably that's a better way to phrase it yeah um so in terms of the way Sherlock Works our claims process is fully on chain fully trustless Sherlock I our team doesn't have control over whether a claim it gets paid out or not that's different than off-chain insurers or insurance companies where they tell you whether your claim gets

paid out or not so we think that's a really important part of D5 and why D5 crypto is so cool because our entire claims process is in smart contracts and basically there's two committees to it so first committee is a four of seven multi-sig of some Security Experts in the space and I think there's a Sherlock person on there as well and they can decide okay we think the hack was due to Sushi which you know whatever is not covered some situation where it's not covered or it's due to some situation that is covered because the the audit that Sherlock did was messed up um that's the first committee if that committee votes against the protocol and says it doesn't get paid out then the protocol can escalate their claim to the

Uma optimistic Oracle Uma optimistic or Oracle is basically thousands of Uma token holders so uma's a totally different protocol their their whole um you know purpose for being is to be a third party sort of Judge system um and so their token holders get to vote on whether the claims should get paid out or not and that way it's sort of out of the hands of any individual or out of the hands of that four of seven multi-sig and we can talk about the price to bribe Uma and bribe the Uma token holders and it has to do you know there are some economic guarantees around uma's market cap but to keep it simple four of seven multi-sig is deciding first and then Uma

token holders decide second Sherlock has no relationship or doesn't own any Uma tokens um so we think that that's a very unbiased way of deciding whether the claims should get paid out yeah so um sir one more question is um where's the where's the the payment coming from is it from the project or do you guys actually uh sticking to that uh about yes so the payment for insurance comes from the project which is different from how Nexus and some other insurers work where it's user-based Insurance you can buy insurance for your position um Sherlock is fully protocol to protocol kind of like B2B I guess um where the protocol is the one deciding to buy the insurance and paying

for the insurance and filing the claims understood thanks sorry one more question how long is the the insurance do you have a annual premium for that or like your auto insurance um well because it's on chain we actually have to get the sort of funds up front and so you basically pay money into the contract and we slowly draw it down over time so you can put three months worth in there you can put a week's worth in there and re-up it every week you can put a Year's worth in there and re-up it every year you can also take it out at any time I think we uh we don't allow you to take out seven uh days worth of Premium so that basically enforces that you have to give us a

seven days heads Up Before You Exit um but yeah you can kind of manage that as that cash flow as closely as you want to so where's the money that I guess claims would be paid out coming from um so Sherlock has its own proprietary pool of LPS or not proprietary it's open but it's controlled by our protocol so LPS institutions VC funds individuals are able to stake into that pool contribute Capital they get the premiums from protocols we also invested into some yield strategies to get them some extra apy but to answer your question it's uh basically anyone in D5 can

contribute to a pool of capital and then are any of those people involved in the 407 multi-cig or I guess are you aware that they are large stakeholders in the Uma protocol none of the large stakeholders in that pool are on the 407 multi-sync cool cool I think uh seems to be all the questions or anything more questions awesome well that's that's all I've got

for you guys today go to sherlock.xyz sign up to be a security expert see if you can make some money and uh yeah thanks a lot appreciate it thanks