The Current State of Audit Contests

Transcript

so much all right hey everyone today we're going to be talking about the current state of audit contests um like I said some of the biggest protocols in the world are using audit contests as part of their security stack to make sure their protocol is safe before going to mainnet and on the other side there are hundreds if not thousands of security researchers who are now making a living earning extra money on the side by participating in these open audit contests and securing some of the most important code in the world while they do it just out of curiosity how many people in this room have participated in

an audit contest before wow like probably 30% of the room um so as you can see it's a very important part of crypto security if you're not familiar with aut contests and how they work they kind of fit somewhere in between traditional Audits and Bug bounties um traditional audits have two to four security researchers audit contests have usually hundreds of security researchers coming in and participating trying to find bugs and code and they get paid based on the bugs that they find so the higher severity bug they find the more they make out of an audit contest pot

everyone is competing for the same pot the more unique the bug is is so if two people find it rather than 10 people that bug is going to be worth more those people are going to get paid more um and as you can see here you have similar structure to an audit where you pay the full amount upfront um and that amount is put into a contest pot mainly paid out to the security researchers who find the bugs that are most impactful to the protocol and most unique um there is a salary paid to security researchers for for a traditional audit on the audit contest side of things there's no salary but there's a fixed pot of money that everyone can access and on the bug

Bounty side of things which is usually something you put live once you go to mainnet um there's no fixed amount of money at all so you only get paid if you find a bug during a bug Bounty and protocol teams only pay out only pay any kind of money if a bug gets found in a bug Bounty program um bug Bounty programs usually last as long as a protocol is live on chain so it can be months it can be years whereas audits audit contests are usually going to last a few weeks um not everything on this slide is exactly accurate in terms of audits and audit contests and we'll talk a little bit about that um further down the presentation so how are aut contests

going how popular are they are they these days as we saw probably 30% of this room has participated in in an audit contest which is awesome and crazy there have been 625 audit contests in the history of crypto pretty much um 336,000 close to 37,000 vulnerabilities have been found and almost 15,000 higher critical vulnerabilities have been found thanks to audit contests $36 million have been paid to the participants in audit contests I know there's multiple people who have become millionaires just from participating in audit contests and we've seen a huge influx of security

researchers um upand cominging and experienced participating in them because of this crazy opportunity to make life-changing money in a lot of cases um by participating so how's it going in 2024 we're only you know 10 months into 2024 let's call it there have been 221 audit contests it's a new record uh for audit contests this year so you can see it's thriving 800 8,000 vulnerabilities rewarded in just the first 6 months of 2024 alone and 3,000 higher critical vulnerabilities rewarded in those first six months almost $10 million were paid out in the first six months of 2024 alone

and that's going to be even bigger as we get to see the full year a lot of audit contests haven't paid out even if they finished in October so we're waiting on the data but there's going to be probably close to $20 million in rewards paid out just this year so some recent data October 2024 that was last month we had 26 audit contests across all platforms that's in comparison to 19 a year ago and I really wish we could see the number of participants across the entire landscape of audit platforms um but we can't see that so I'm just going to show you Sherlock's data um I'm the co-founder of Sherlock by the way if you didn't um see that

earlier we have about double the amount of audit participants in audit contests last month versus October a year ago which I think is really awesome and shows that the space is growing security researchers are are getting um up to speed are arming themselves learning about new vulnerabilities and I think it's really contributed to that slide that Peter catcher ginsky showed at the beginning of this session that essentially not that many bugs in smart contracts have caused losses compared to last year which I think is really great so I would call overall the current state of aut contest very healthy things are going in the right direction all-time highs in a bunch of

different areas and this year is looking great overall in terms of audit contests of course there are some things we want to be aware of and I want to flag one of those things today and talk a little bit about just the different directions that audit contests are going in where the space might go in the future as we saw on this original slide um it's not exactly accurate what we're saying here in terms of audit contests because there are so many different types of contests and there's a few new trends that have come out in the last couple months in terms of how audit contests are being conducted so some of those new trends

are fixed pay for security researchers to participate in audits this is something that's not that new we've actually seen it for a while now where Security Experts are getting paid fixed dollar amounts to show up to contest and this is very much like an audit you know if you're doing an audit you're getting paid a fixed amount you know you're going to make money during that time you can dedicate all of your time to looking at the code which is great um another area we've seen is a variable price for protocol teams so fixed pay for security researchers looks more like an audit but a variable price for protocol teams looks more like a bug Bounty where the protocol team doesn't actually pay out until a bug is found or a certain number of critical bugs are found variability

and contest length audit contests historically have been scoped a little bit like audits where you want to give every researcher enough time to look at every single contract but we're seeing some variations in contest length where maybe there's a ton of code we I think we've seen a 100,000 lines of code and you get like a month to review it which is definitely not enough time to review 100,000 lines of code but it's more like a bug Bounty where people say look you can look at it you can find bugs that you can find and then whatever you find in a month will pay you out for that we've also seen no guaranteed payouts so that's the same thing as variable cost for protocol teams but for the security researchers so you may not get paid in an audit contest unless you

find a medium severity bug or you may not get paid in an auto contest unless you find a high severity bug um and we've also seen a little bit of first come first serve findings payouts where the first person who finds a bug during an audit contest gets paid more or even in some cases gets paid everything and that looks a lot like a bug Bounty so we're kind of seeing a trend where there's almost two types of audit contests that are being created now or it looks a little bit like there's two separate types of audit contests you have one which looks a lot like an audit so they can be open or closed they have reserved security researchers there's a fixed cost for the client fixed pot for

the security researchers the length is based on how long it takes to actually review the code these are all um similarities to a traditional audit or a lot of them are similarities to a traditional audit and then we have this other type of audit contest which is totally open but there's no guarantee around what security researchers are going to show up there's a variable cost for the the client the client could essentially pay nothing and there's unlocking vulnerability tiers a lot like a bug Bounty where you don't get paid unless a certain severity vulnerability is found and the length of that contest is often decided by the client less how long it takes a security researcher to review that whole code base more hey

this client wants to launch in three weeks so the contest is going to be two weeks long because of this I think there's two types of audit contests now or two types of contests sorry there's an audit contest and there's bug Bounty contest and these are actually very different from each other and I think it's worth separating them out and kind of understanding what a client is trying to achieve with their audit with their contest whether it be an audit contest or a bug Bounty contest um because doing an audit is very well I'll get to it on the next slide but doing an audit is very different than setting up a bug Bounty program and has completely

different security implications for a protocol team so audit contests and Bug Bounty contests both of them are extremely valuable in terms of having arrows in your quiver you want to be able to have both you want to be able to deploy one or the other based on the situation that you're in if you're looking for an audit a really robust audit an audit contest is a great choice if you're looking to do a bug Bounty program right after you finish your audit auditing um or go to mainnet very quickly then a bug Bounty contest can also be a good choice and can help you save money as a protocol team instead of paying out three million do bug bounties you can do a $1 million

audit contest and you get all three of those bugs potentially in the same contest uh which saves a protocol team $2 million in that case um so both are very valuable and both can be used depending on the situation however audit contests are essentially a type of audit um bug Bounty contests are not bug Bounty contests do not guarantee any security researcher participation they don't guarantee code coverage obviously an audit doesn't do that either but there's an expectation that the full code is reviewed there are dedicated security researchers whereas in a bug Bounty contest it's less clear exactly what you're getting in terms of um security coverage we'll say so I think one thing

that we should be aware of as different platforms are offering different types of contests is that we need to be very careful about how we use the term audit because audit has become kind of a restricted word in the security industry and we don't want to be selling a bug Bounty product and calling it an audit because I know a lot of people in this room have put in a lot of work to making sure that crypto audits web3 security audits are valuable are are doing full coverage as Peter kater ginsky showed on that slide we're going in the right direction so I just want to make sure that we aren't going to start confusing people by calling bug Bounty contests

audits because bug Bounty contests are definitely not Audits and this is very important so overall the state of audit contest very healthy going in the right direction top protocol teams are using them they're a great tool to find critical bugs before you go to mainnet before you put up a multi-million dollar bug Bounty which many teams are doing now uh but we need to be careful about it it's a very powerful tool and we need to know exactly we need to tell protocol teams exactly what they're getting um when they sign up for an audit contest and what that audit contest is achieving whether it's an audit or whether it's more like a bug Bounty so overall both paths are great whether you choose to do an audit contest whether

you choose to do a bug Bounty contest we're headed in the right direction Things Are really good but let's make sure to keep the bug Bounty terms dedicated to bug Bounty products audit terms dedicated to audit products and that way protocol teams will know what they're getting and we will have a lot of safety in defi thank you very [Applause] much any questions overall state of aut contests bug Bounty contests anything at all uh yeah sure thank

you is it me or uh yeah okay on the couch uh so one question I have is like as um as an AA developer I am one of the Developers of the other protocol we have done many uh back Bounty programs and campaigns and a variety of of different um activities around like the security of the protocol so one thing I noticed is that uh it's really hard sometimes to discern uh reports that are actually you know valuable and meaningful from reports that are maybe attempt to get a very easy Bounty or reports that have

been reported multiple times yeah so there are a variety of challenges around uh participating into these campaigns mostly because they take a lot of mental effort in trying to discern stuff that is actually meaningful and valuable from stuff uh that is uh less valuable or maybe just plain and simple waste of time so do you have an idea about how moving on in the future we can actually challenge uh this uh state of of things and maybe improve so that the experience get even better yeah great question so in an audit contest they're totally open you can receive vulnerabilities from anywhere basically anyone can submit vulnerabilities which means that you can

see everything you see people using chat gpts to submit things that look real but aren't real basically you know like every type of thing under the sun you can see in a submission in an audit contest um so there are some things that a lot of platforms do like only allow high and medium severity vulnerabilities so that all the fluff underneath isn't really a part of it isn't really rewarded isn't something teams have to look at another interesting thing is we've seen over the course of you know two years of doing audit contests now that judging is really difficult and I think that's kind of what you're getting at is how do you discern between something that sounds like a bug versus something that actually is a bug or

something that isn't that useful to the protocol team versus something that is and you get all these different um submissions and so I think we've seen platforms um moving towards dedicating a higher dollar amount to judging because it's become clear that you really need to have somebody who can discern well between what is a bug in that specific type of protocol for that team and what isn't and this is a really difficult problem I don't think there is a fully objective way of always determining that so having a great judge for those audit contests is the best way we know how right now maybe AI in the future like training against like pass submissions and stuff could be an idea just yeah I

mean we've got AI doing submissions so we might as well have ai doing judging exactly against it they cancel each other yeah yeah uh I think someone over here had a question so my question was similar to what you talked about just right now uh about the uh lows and gas optimizations not being awarded too much uh so in the beginning they were and now we have moved to a situation where only mediums and Highs are being awarded and do you see a future where uh only highs and criticals are being awarded and uh if yes do you think that this will be uh some kind of uh an entry barrier for beginners who are uh looking for these

low hanging fruits and uh low and mediums sorry entry barrier for who uh entry barrier for beginners for amateur security researchers yeah yeah it's a great question um I think one of the great Parts about audit contests is that you can start out not knowing a ton about security and you can still make $50 in an AIT contest you can make $500 I mean a lot of people in this room have made $100 $11,000 $3,000 in an audit cont contest and said that's it I'm quitting my job I'm dedicating myself to this fulltime which I think is amazing and awesome um but it is true that what's valuable to protocol teams

especially as they get close to mainnet is more on the criticals the highs and they care less about hey you can change your code in this way to optimize something because they're like look we need to go to mainnet we need to just be secure at this point so there is a little bit of tension there between having Auto contest that accept all kind of vulnerabilities and making sure that upand cominging security researchers can make money gain confidence you know build momentum in their career by doing audit contests which I think has happened a lot um versus balancing protocol teams who in certain situations may only want high and critical severity vulnerabilities before they're going to mainnet so I don't know the exact answer to that question but I hope we'll see a

balance where beginners can still have the right on-ramps to get to become top Security Experts in the space like we've seen over the last couple years thank you uh yeah is it safe to say that there's more incentives and Bug bounties than audit contests I think if you look at the dollar amounts the dollar amounts are a lot higher in bug bounties um but the likelihood of getting paid is much lower so it's or in most cases I would say is is lower yeah

sure so would it be better to start off with audit contest and then go into bug bounties 100% 100% I think we've seen a lot of teams take that path especially doing you know more traditional consultative audits on this side audit contest here and then bug Bounty especially once you're live on mainnet um because it's it's like a risk reward type of thing so there's a lot of vulner ilities that are going to get found in those early Audits and you're not going to have to pay a lot of money for them in audit contests there's going to be like a medium amount of vulnerabilities found maybe and you're going to pay a medium amount for it and then in the bug Bounty program there's going to be very few bugs found but you're going to pay a

lot for each one of those bugs cool I think that's it we can end the session here thank you guys for attending session 4 hope it was informative and uh yeah looking forward to session five