The Confusing World of Smart Contract Security

Transcript

we love that here so I'm going to keep it going because we want to try to get back on track with our our next speaker uh who's Stefan Byer from from solidified and his talk is called the confusing world of smart contract security so give a round of applause for Stefan everybody okay so this talk is going to be relatively similar to the previous one there's going to be a few difference of opinions but which we should discuss afterwards but yeah um so this is called the confusing world of smart contract security and I'll talk you a bit through the different options companies have for um Contracting Security Services and also what you can do before even thinking

about getting an audit you know what you can do as a project internally so just quickly about me and about the company uh I'm from a distributed systems background and at some point I went into cyber security and then decided to combine the two and I now run a company called Oak security which has two Brands one of which is solidified which looks at the ethereum ecosystem solidified has been around since 2017 and is one of the oldest audit companies uh around and we've audited a great number of projects over the years and I've developed a methodology which is somehow unique um but anyway let's go back to 2016 uh which is when Security First became apparent as a problem in web 3 everyone

well most people here will remember remember the Dow HCK or exploit the younger ones might not but it it was sort of the first time where smart contract security became a problem for the via community uh it affected 15% of all ease in circulation at the time which was obviously important so I'm not here to talk about the Dow exploit anyone who doesn't know about it can can read about it there plenty of of sources but it it's also when security audit services just became a thing right people started founding companies us as well to to look at this problem and then fast forward to last months June

2023 and there were three reentrancy exploits the same exploit of the Dow HW in three big projects according to this list here U which is a good source of of anything related to re-entrancy attacks that shows a number of things but it shows that in seven years we've not really moved on in smart contract security right we we still have the same exploits we've got other exploits as well but but there still a problem around similarly if we look at the wed leaderboard which is a good source to look at everything that happens in in defi or in in in web 3 you can see the top 10 here Snapshot from 2D of July there of the top 10 hugs or exploits

eight projects were un audited right and the other two were audited uh so that's interesting a lot of projects are una audited and another interesting thing is if you want to if you were to go down this list to 100 or more um we'd see a lot of names of audit companies listed there as well right so so even the audited projects are not safe and that's a very important thing and that's what I what I sort of you know where I disagree a bit from the previous speakers the skin and the game idea there's only so much an audit can do right uh there are and and I explain a bit why in a later in the slide but uh the auditor does not have the full responsibility for hug there or exploit there are other things

very often um that can go wrong of course there are incidents of of Auditors doing really bad work so what is an audit first of all right so an audit is a Time booxed review by Security Experts you get a number of people applying a number of techniques in a a certain methodology or process every company has their own way of doing things and they look at uh and they you know they return a list of problems in in a project and that's very efficient at detecting certain problems right so a number of findings are very good for Audits and type of exploits and their vulnerabilities that a team usually

doesn't detect but an audit does uh just because externalize a methodology being applied and and someone working differently um but on the other hand there are certain things where an uded is very bad uh at at finding things there are bugs that are very easily found in fast testing or in in any type of testing and very difficult to find in a in a code review and that's where other things have to come into play right so not it is just one slice of this famous Swiss cheese approach to Security in Al alternative to audits as mentioned by the previous speaker are audit competitions and an audit competition is like an open audit where

more people can sign up U anonymously not so anonymously however they want and they can uh provide findings and there's a certain budget which is shared amount anyone who who among among the the people that have findings um we compare the two in a minute but just to complete things there are also buck which are sort of a last resort you put up a very high price tck to basically convince people to look at your already life project and Report bugs instead of exploiting them and maybe even convince a black cut to turn white cut and not not run an exploit you know if you've got a 3 million exploit possibility and you've got a 1 million Buck

Bounty it might be easier to take the million and not go to prison or not risk going to prison so that's sort of the idea behind it um so which is best right so the the previous speaker had a clear opinion on this and um I sort of say it depends right I think the previous speaker also agreed that traditional audits have a place uh the difference between a competition and the other two sorry between an audit and the competition and back bound is that it usually goes deeper right when we pay our Auditors to look at the things we choose a professional team applying certain methodologies and they look at every line of code uh at a certain price of course um but you can be sure everything is covered

and if you've got a serious provider you get very transparent overview of who's auditing your code how long he spends on the code and and and what they find and what they do and what methodologies I apply right if you don't get that you know if you don't get the background of the people working on your code or the number of hours or or or engineering weeks invested in this that's a red flag but anyway in a competition you get much more eyes but they don't look as deep usually and we know that because we encourage our Auditors and our um freelance uh participants to to also do competitions and back bounties we consider a good training for them and and they like that um but they all agree

they don't go very deep because there's uh you know particularly if a project is very bad and there a budget is meant to be shared amongst a lot of people it's just easier to move on after a certain amount of time so they look for the obvious findings they report them and then they move to the next project so they don't go as deep they don't cover every line of code it's like an 80/20 approach um and the same goes for back Bounty right uh people don't spend a month on a back boundy usually uh working full-time they you know look if they can see anything if they can't they move on to the next one and obviously a p Bounty also has a disadvantage that it's very expensive ensive right if you have a million doll

back Bounty you have to have a million dollar in reserve uh to um to make sure you can you can pay it out and it should be a last resort you know like uh that million you pay out for critical issue should be less than the exploit uh the potential exploit so now I want to talk a bit about our approach um obviously I believe in this approach but the idea of this is not to sell you that this is the only and the bestest approach it's just to give you a bit of an Insight what goes on and what type of questions you should ask when you talk to us or any other provider and there are plenty of good ones um so our process has been developed since 201 uh

16 17 uh depending on if you take the start date of the company or when we first started thinking about these things and um basically we have a pool of Auditors and we Lally choose them with very different backgrounds and and whenever we get an audit we make sure every line of code is covered by at least three people probably more and they are chosen differently to apply different methodologies of course we have our Global process but within that they are free to choose their methodology and we make sure these methodology are complimentary right they're not all doing the same things because that's important because now for example it's it's become fashionable to get a solo audit right A lot of people on Twitter selling their service as a

solo audit aor and that's fine but you have to bear in mind you get one point of view one methodology one set of eyes different people thinking differently with a different background find different things we also make sure there's a the background is diff different and M and matches the project right so if you have a defi project with an economic component we always make sure there's an economist on the uh on the team we don't just employ computer scientists or security well they're all Security Experts of course but uh they they have a background in economic modeling some of them um and then when they get to work they work blindly that's a big difference to other companies blinded or blindly whatever in the sense that they

don't share their findings with each other they talk to each other of course discussing the project uh architecture design but they don't um share their findings until a certain point in the audit which we call the consensus meeting and the reason for this is obviously to keep everyone on their toes right nobody wants to be that person in the consensus meeting that has missed a critical issue that that you know that's usually bit of an internal competition but it also uh stops people from biasing each other and importantly it gives us points it gives us data for quality control if we have a lot of overlap uh then we are relatively good usually if there's not enough overlap we have to look into the project look at what went

wrong maybe offer training to people that have missed something and and try to make the team stronger this way and and everything is transparent right so our clients get a report and the report is always made public and everything is discussed so they see the Auditors they know who they're dealing with they know their background and um I'm not saying this is the best way but of course I believe it's the best way because we work because that's why we've chosen it but it should give you ideas what type of questions you should ask when you talk to audit service provider right like who are these people do they cover economic aspects of my protocol do they look into uh for example if it's cryptography if there's a CK roller

processor for example do they understand how the CK stuff works is there cop on the team and you know how do they work what other methodologies involved do they write tests or do they just look at the code uh you know we do both we have people doing formal methods and we try to mix and match anyway so this is all all good um but I said before projects have to take responsibilities themselves because an audit is only one slice in the Swiss cheese thing uh what can projects do before an audit or or during an audit or even in during the whole life cycle so there are a lot of things that can be done in terms of

tooling right particular in the ethereum world there are lot of tools available testing is essential unit testing 100% code coverage or at least High code coverage that makes sense 100% code coverage can be very bad code coverage you know you don't necessarily find everything just because you have 100% code coverage uh test coverage uh you can look in things like Fast testing should always run uh static code analysis right tools like Slither are very good at finding the obvious stuff uh I always say these tools are good at finding the type of exploits we had in 2016 2017 so re-entrancy possibility I usually highlight you don't have to have

a an issue with external calls and re-entrancy in an audit that's something that a good team can find by themselves before an audit anyway there are a lot of tooling available H okay you should also look at you know other things so smart contracts are are only a small part of a project there there are other things involved like Key Management you you can pay 100K for an audit and then upload your private key onto GitHub right we've seen that you know teams just um having key leakage or or a bridge having the the the hotkey not secured that sort of thing there's a lot of offchain code involved usually your apis

your user interfaces you know a lot of projects get their DNS hijacked instead of uh their smart contract hacks and they just you know diverted to different user interface interacting with different smart contracts or not even with smart contracts right um there's humans involved users and employees do very weird things right you have to try to anticipate these things you have to have policies for it you have to have policies for what do you do when things go wrong what do you do when a key is exposed how do you re revoke that how do you issue a new one if you have a plan for all these things it's much easier to deal with it and finally this is the last slide um usually when we're dealing on with web 3

it's Financial software right and that's critical software so a lot of teams just don't realize that they are building Financial software it's um Financial software means you need someone with a financial mind on the team right so if you're building Financial software get a finance guy on the team right get someone who knows the finance system don't just invent a new economic model and think it's just going to be work just going to work and they're now ways to to game it uh Game Theory type of economic exploits are very big thing um specify and document everything that should be clear right you can only implement the model correctly if it's if it's specified and documented right your

your developers won't be able to implement uh the finance gu ideas if they're not written down carefully or correctly uh slow down you know do everything carefully test everything uh keep it simple the you know the type of code NASA would write to send a spaceship out that's the type of code you want to write in a smart contract for defi protocol um and you know just don't innovate everything at once just you know the new economy can be built in in small steps and you don't have to reinvent everything and just expect it to work um so that's just my advice for projects um you can find us at our website and on

Twitter and of course you've got me here for for [Applause] questions yeah sure this oh hey um hello okay thanks for your um great talk um what I want to ask you is so obviously the state of security right now in web 3 and U notably in um defi is not good right I think we can all agree with that so my question would be like what is your perspective as an auditor how we need to innovate or maybe as well

how Auditors need to innovate to that we as a ecosystem become better I I understand your view right it's not all on the auditor it's not you know like the auditor is not the only one responsible but I still think we need to innovate as well in the auditing space because somehow we have to overcome this challenge right yeah and and that's right and I completely agree with you that competitions are a good way to do that uh and as you I think there should be a second step because you mentioned this problem and that's what our Auditors when they go to competition competitions they complain about you know that that if the project is really bad they paid less because it should be the other way around right um so so competitions are good commentary approach and

uh we I don't want to make an announcement here but uh we are exploring ways of combining these two things and maybe we should sit down and talk on how we can work together on this um and obviously another thing that the you say the state of security is bad and and it is true but it's becoming better uh one of the problem is that everyone grows too quickly and too fast including auditing companies uh I mean we we've been around for a long time but we've grown very carefully we deliberately rejected a lot lot of work because we just didn't want to employ 100 Auditors that were not you know the best so we we have they liberally stayed small and and some others might not have

done this I mean you mentioned the company before and I I listened to talk to them last year in August where they said they had done 1,000 audits that year right there's not enough security engineers in the world to do our methodology in in 10 years for to reach a thousand audits right so it's people cow very quickly and want the cash thank you for this interesting talk uh so I had a question so you said you guys adopt a blind auditing um approach so wouldn't that uh mean it would be better sometimes to employ separate Auditors from all over the world not from a company to be auditing the the

project so that they're not communicating or have no um way of communicating together yes that that's another way of doing it um we do that internally and and uh and we quite often I mean the bigger projects they often employ us in parallel of trial of bits or or consensus diligence or someone like this or one after the other and we sometimes get to talk to them sometimes not but it's always interesting when um when you compare findings but uh here the advantage is you know that you get a single report we already fight it out between each other what really is an issue and we we we particular managed to remove a large number of false positives because if you know if you're a single

guy or or or a team looking at the same thing and you all bias each other then it's easy to to just convince everyone there's an issue and if I've missed an issue that you have found I I'll make sure I look very deeply into this to make that that it's not a false positive so we we we deliberately manage to to get a very concise result for the client yeah uh thank you for talk um the question that I have um is yeah based on the talk it seems that everything boils down pretty much to understanding if if the Auditors spend time on looking on your

code and like you know doing what they're supposed to do so I'm I'm curious if you're like um exploring or at least like look at this concept where the projects injecting the bux into their codebase just to just to make it to serve as a kind of proof of work if the Auditors did their job if they spend specific amount of time to detect those vulnerabilities maybe you experience this during your audits or or yeah yeah we've experienced this and uh several times and we prefer that if we know before that it's done um because sometimes it could be seen as a way of the client to get findings removed from the report so one nice way of doing this we've seen and we were very in favor of was a team just having an internal audit

it then they left certain bugs that were found in the project so they were real bugs not uh not artificial so they were not uh you know not silly things that you just find directly easily and um and then they generated a hush of all the descriptions of this and published this so so you know we could they could later prove to us that they were really injected bugs and that worked really well and uh we we actually did very well which which was sort of a relief but it's sometimes you know you oh this is something we did on purpose and it doesn't really look that way you just don't it's embarrassing and you don't want it in the report right so that's because the report is public and it says fixed but it still says what was found yeah okay

great I I think you touched something super interesting uh like audit report cleaning and um so I I hear that like many times that br projects have an issue with um getting an order report and many issues are being found and that there's kind of like an incentive to uh remove these findings from the audit report um I personally I don't agree with that at all I think like why would be a project ashamed about it yeah like um but what what is your take on like audit report cleaning and how to tackle it and what kind of the what's the M motivation for the project or how can we motivate them to understand that it's not a bad thing to be transparent about it and it's not shameful yeah it it's sometimes a

difficult conversation to have and um I mean we we we our policy is if it's extremely bad we will tell you before we even write the report and uh then and you know if you decide to rewrite everything and don't publish the project then we don't have to publish the report right but usually we try to convince people that you know having the report public Is Not a Bad Thing even if there's critical findings in there and theyve been fixed you know it just shows that you've done your due diligence um very occasionally we we get clients that are not happy and they usually the clients where we already doubtful if to take them on or not you know there sort of some red flag and and sometimes they

they they fall through our due diligence and we have to have an very uncomfortable conversation and and what we always do we allow each team to write a reply which we ritim copy into the report so our reports are published as a creative common non uh non-derivative attribution thing so you can you can do with it whatever you want but you cannot modify it right so we we we own the copyright of that report and we add every it's in our contract that they can add a a client reply to each issue and we publish that so if they don't agree or want to give an explanation why it is you know sometimes they put things like oh this particular contract was submitted to the audit too early it hadn't gone through internal auditing

and if they want to do that they can but for the good teams it's usually not necessary worried abouts yeah yeah I mean we've never had a report where we didn't have any findings uh it is sometimes you know you have a very good project and you have three informational and one Min are finding and you you sort of published this and there's question you know why didn't you find more well you know it was really good you know if you if someone you know some of the big teams they employ three security gu guys internally they they already had maybe had another audit before they had done a formal verification program so so it's

sometimes difficult but but yeah again the track record of the auditor usually speaks for itself you know if you have an auditor that is on this wed 100 list I mean it's very easy to get on there right we are not on there but we might be one day I mean a part of this is luck right but if you're on there in 50 out of the 100 top 100 entries then and all your F Reports say no findings then there's an [Applause] issue