State of Audits

Transcript

I'm Ryan I'm an engineer at Electric Capital and today I'll be walking through the state of audits so smart contracts are are unique in the sense that it's kind of one of the first cases where we have all of the production code predominantly open source and transparent and running on chain so what you get is this massive graph of cross-contract dependencies and Rich data that is really valuable for both users and security researchers however one of the difficulties is publicly linking the con the contracts with the audit histories which actually we believe will make protocols safer to

use and easier for security researchers to dig into potential cross-contract impacts but today there's a problem with the audit space because there's a large gap between Audits and deployed contracts because in the workflow of Engineers and smart contract developers today audits happen pre-deployment but most of the users what they actually care about is the contract that's on chain and linking the gap between the deployed contracts and the audit reports that were done pre-deployment is extremely expensive and we actually find that a lot of security researchers they spend the majority of their time reconciling

commits with deployments instead of looking for bugs so as a result we we went out and we collected pretty much every audit report on ethereum since 2018 in ethereum D5 and we open sourced this in the first version of the crypto audits mapping we collected other reports of a representative sample of ethereum D5 representing uh greater than 92 percent of ethereum tvl since 2018. but just collecting these other reports is very challenging there isn't a very clean or easy process to do so because if you start off at D5 llama what ends up happening is you end up going to the protocol website sometimes you go back

to the auditor website but eventually around 71 of these audit reports end up living on GitHub but these other Reports most of these other reports today are in an unstructured format they're predominantly PDFs which means parsing them and extracting information valuable information from these other reports is actually quite costly so we tried out using so we built embeddings across these PDFs using gpt4 and we subsequently asked a series of classification type questions such as what is the name of the auditor can you give me the critical risk findings but more importantly for us we were

really looking to link the audit reports with the contracts that are deployed on chain and more importantly the repositories in themselves one one of the interesting findings that we found uh which is should be pretty common sense is that the larger the tvl the protocol uh the more audits attend these protocols tend to get with protocols larger than 1.1 billion in tvl averaging around 10.6 audits with a median of around four audits so protocols in the space are are get are are using audits but we think that there is still this gap between the other reports in themselves the GitHub repositories and the actual deployed contracts

luckily most other reports have references to a git repo or commit hash so that solves the first link but it gets a little bit Messier when you actually try to as a user of D5 protocols to link audit reports with the actual deploy contracts because one thing that tends to happen is you tend to get a lot of audit drift between the time of the audit and the code that's actually deployed with only with less than two percent of audit reports actually referencing the deployment addresses and that's really concerning from a user perspective and it really highlights this large gap in the current workflow between audits happening pre-deployment but most of the users only really caring about what's

actually deployed on chain because that's what's actually custody in the value and as a result we kind of see that most users instead of referencing audit reports or really reading the audit reports in themselves they end up falling back to a measure of lindiness of a protocol to determine if it's secure and whether it fits secure enough for them to put their money into so a user might look at a chart like a tvl chart like Lido and say okay this is a protocol it's been around for two and a half years it's custodying 15 billion dollars if I'm putting in 10 million 25 million probably okay because I'm assuming that everyone else has done their homework

and this is actually the thought process that goes through a lot of the users in the D5 side because it's so expensive and difficult to make the tie between the audit report and the actual code that's deployed on chain we can we can view smart contracts as custodians of assets and live bug bounties smart contracts either are directly directly custodying assets for example an LP token a lending protocol a yield management protocol or they're indirectly customing assets through wallet approvals and because these contracts that because the assets are a massive bug Bounty the

longer the implementation of these smart contracts is lied for the more Lindy they become but the most important thing is that what we're trying to come up with a metric that defines how Lindy a certain protocol is it's important that we reset that Lindy metric whenever a smart con whenever the protocol upgrades its smart contracts and they actually do quite a lot so here's a simple example of a protocol that that launches at time Step Zero gain some TBL kind of flat lines but issues to upgrades from a traditional user's perspective they'll look at that TV outline on the left and they'll think okay um this protocol has been around for a

while like they're gaining tvl and but but the problem is that with the upgrades the lindiness of the protocol actually resets to zero so let's get into I guess a bit of a spicy territory let's walk through an example of a D5 lending protocol most users will look at a chart like this things seem pretty good this protocol is custodying 100 to 200 million dollars in assets that kind of equates to a bug Bounty of 100 and 100 to 200 million dollars that's been live for a year and it looks pretty good but when we actually dig into the code so here I'm using Code slaw.app I can dig in find the smart contract and I

realize that it actually implements proxy logic that's actually not standard ID code the relevant storage variables from the contract and I pull the history so for this I'm using tokenflow.live which is uh evm data provider and what we see is that this protocol actually has a lot quite a lot of updates at the top of the table you can see the initial deployment as the as the storage in the contract is being set from zero to the to the initial logic but then subsequently you get a lot of module and proxy upgrades because protocols do upgrade quite a lot especially if they're very early stage and if we overlay the upgrades with the

tvl we can see we can start to see the issue of just relying on the TV all line as lindiness so you can see that near December of last year there was an upgrade that subsequently led to the exploit but even after that upgrade was pushed another 150 million dollars went into this protocol before it was subsequently exploited and then they paused the protocol by issuing another upgrade most users will just take the line on the left the integral TV all as perceived lindiness but the security actually resets at every proxy upgrade giving this more Jagged chart we don't want to give credit when there's new implementation logic but we think that there is a lot of room

for improvement on the audit side and that the current state of audit reports and the lack of linking between audit reports and deploy contracts is not the future we think that the future of audits is probably an audit metadata protocol this isn't a new problem that the web hasn't faced before and they're examples such as robots.txt or the open graph which are examples of protocols that have standardized previously unstructured metadata to improve user experience so in conclusion uh we collected every other report on pretty much every other report on ethereum D5 very few of these audit reports actually link between the actual code and the deployed contracts

but we think that these two first two issues are solvable with a metadata standard but in the interim we think that users most likely unfortunately end up falling back to derived measure of lindiness but we think that one way to get around this is by exposing and making more transparent the prevalent nature of proxies and upgradability that is very prevalent within within D5 and we propose stvl which is a metric that resets the lindiness of a protocol whenever the Logic independencies upgrade thanks [Applause] [Music] um