The security conundrum - building the plane as we audit it / Anton Astafiev

Transcript

thank you hey everyone um here's the clicker cool hey everyone I'm Anton I'm headed security at metal Labs the team that is behind the key sync light and the key sync era but today we're not talking about player 2 and which advantages roll up Spring and whatnot today we're talking about how to build a secure system build it fast achieve the reach the market get the adoption without actually compromising the security which is a tricky question given that the novel technology usually comes with unobvious caveats that we

cannot predict often so a bit about my background I've been working in different web 2 web 3 places in development in privacy and security some research and so on most lately I'm involved with different blockchain projects and doing security for blockchain um basically what we're talking about today is how can we align the speed of development everybody wants to move fast the product team wants to move fast developers want to roll out their stuff as soon as possible and at the same time taking a as careful approach as possible doing everything that is in our um under our control to actually protect

the users to not ship raw systems to not really allow unsecure code to reach the production and to lead to a terrible compromise um today I will be talking about how we approach this and this is not going to be very metal apps or is it specific this is going to be more of a how to really juggle those priorities so very important disclaimer first this talk is not for the people that are looking to cut Corners this talk is how to speed up the time to Market without compromising on security without cutting

Corners if you're looking how to cut the corners is not the right talk and a very important thing if you are looking to ship the secure system the Audits and all the security measures will take quite some time you cannot really shorten the time that it takes to secure the system you can only juggle it in a different ways to um reduce the time to Market so here's to our story um our heroes are security team development team product team and guest staring is the external auditors now we're switching on to the problem and this is going to be an overly

simplified or slightly exaggerated conversation that I unfortunately have seen during my career several times and the way it goes the development team is like okay we're almost ready the code is ready tomorrow and then the product is like Yay launching launching launching are the audits complete and I don't even know where to start to explain that you cannot really have completed audits if the code is only going to be ready tomorrow you this is just not possible um slightly exaggerating but only slightly this has happened in different forms and shapes some several times through my career so what can we do in the worst case scenario again overly simplified let's say we need around two years of

development worst case scenario at the end of the development like okay good we are ready we're ready for the Audits and the auditor is like yeah you know I'm available in a year I think everybody's been there and then the audits also take a year and time to market for years and the product team is like what and the investors are like you said you need two years to develop the thing why are we now just picking double the time this is yeah this is not what we agreed on and what is what is going on here so here we can start with a very obvious thing like I think everybody is doing we'll remove this waiting this is the obvious place where we can cut

so we're eliminating the waiting time there are two ways to eliminate the waiting time one is pick whoever is available right now and just you know kind of compromise slightly on the quality of the audience this is not the recommended way we really do not want to cut the corners we need to engage early in a better scenario that's where we are ending this year one year waiting time is then in parallel with the development and our time to Market is now three years much better but still not ideal however there's an asterisk here what happens if the development is slightly overdue and the development team was like oh we need another six

months you know what happens yeah you know I'm already booked for the next year so sorry next slot and time to Market and we're back to the worst possible scenario so this is something that you need to be very careful about when you're eliminating the waiting time there are a few things to keep in mind first you need to pick your Auditors really know who you want to look into your code ideally this should be the people that really understand what you're building that have the experience with specific Technologies languages and so on and you need to engage early you need to already way beforehand to be in touch with them and to see what their availability is the second thing is you need to really

have a good estimate of the development timeline because you if you are planning the audience one year in advance and your development is going to be delayed or completed faster you need to adjust so the other important thing is you need to when you're already engaging with the Auditors you need to already know at least in some form the scope of the audit it's not going to go well if you come to the others and say you know what we're building an nft platform or maybe defy oh we actually haven't decided yet but are you going to be able to audit it in in a year and you know what it's going to take from two weeks to maybe 40 weeks yeah TBD we'll see you really need to know what you're talking about and

this will also help you to pick the right people and a very important piece here that many people forget is ideally you need to have some flexibility arranged with your Auditors because there may be last minute changes scope slightly expanding things slightly delayed or you're ready a bit earlier you need to be aware and you need to deliver this message to the Auditors in advance and have the agreement with them and ideally have a constant check in with them so we have assigned in a year we'll start the audit but you know what in six months we're gonna have a check-in and see if we're on track are we actually going to be ready by then is the scope has changed has something

different come up in the meantime and these things are the Auditors will be also really appreciating because they will be well prepared to really look into your into your project but the question is can we do better well ideal scenario is the audit is going ongoing at the same time as the development cool and well now we're talking but I feel like we already had this conversation the Auditors are really picky they don't like auditing code that is constantly changing and they really don't like auditing code that doesn't exist so this is not an option however a very good question are you building a model this project or is it modular

I.E dare say that in 2023 if you're building a monolith project and it's larger than a small pet project you're probably doing something wrong because modular development splits into different components come with many advantages like you can develop them in parallel you can test them independently you can scale them independently in a monolith project you're going to have bottlenecks and splitting them out and scaling them out scaling them up separately is going to be very hard and you will spend a lot of time on actually developing this and redoing your monoliths into components upgrading is easier and so on so the general recommendation is split your system and self into self-campaint components which I think many developers

are already doing and build the most critical ones first the thing is how that helps with our auditing approach let's say we have three components that we were developing in three years and in reality it's going to be actually much more complicated they're going to be things done in parallel and so on but let's say we have split it into three main pieces and we're doing them one by one and they take exactly the same time within those two years eight months each we engage with the Auditors at the very beginning we have waiting time kicked off one year and we're building the first component then we're building the second component third component as you can see the audits have started after the components were completed and we can actually

truly and properly audit the code that has been fixed already that has been stable already and this has been completed already so we can say here's the repo here is the commit hash and the Auditors will be able to say this is the code that we audited and everybody will be able to verify that later great and as you can see the development in total still takes two years nothing changed here the audit still that takes two years takes one year nothing changed here waiting again one year however the time to Market is 2.33 or two years and four months great so this is pretty much the main approach that we were taking however there is more just to to do here you see there's like some gaps where nothing is

happening can we fill in those gaps with something and but first let's do this summary on the modular audits first build the most critical pieces first audit them first you will have more time to test them run them on test Nets show to different Auditors run with the community open source the code fully and so on and battle tests those most critical pieces for longer time second ability to move the Sorting days and having this flexibility with your Auditors becomes much more important in this case because now you're not planning just the final delivery date you're planning each Milestone and planning each Milestone is much more complicated and things may be depending on each other

and the most important part that you need to understand this is going to cost you more not because the audience are going to cost you more but because most likely once you complete the first component in our case for example this is the whole layer 1 smart contracts you complete those and you're working on something else that is interacting with those contracts what will happen you will realize some extra features are needed here and there or some things have to actually work slightly in a different way and you will have to update that code and make some some changes into that code making those changes without auditing them is a really bad idea this is exactly how a few well-known hacks have happened like we had six audits of the

code but then we were hacked and then it appears that nobody has audited the latest upgrade where the bug was introduced so you will have to audit your first component and then do a bunch of div audits upgrade audits in the meantime till you actually can launch this comes with extra cost it would be would have been cheaper for you to wait till everything is completed everything interacts nicely all your integration tests work and now give it to the Auditors and they will spend four months for example on it instead of spending four months on it in one run and then two weeks here and two weeks there and two weeks there and you're still paying for it however the price may be quite Justified if we're cutting the total time to Market by by two years

now what else we can do how can we fill in those gaps obviously we want to do internal audits well uh the reason in here Auditors don't like to work on the code that is constantly changing in-house application security protocol security team is embedded in the development process anyway they are involved for at the architecture design stage already I hope if you're not doing this you're you should um they're involved from the very beginning they know what's going on they know what the system is doing they are participating in the design decisions they are contributing to those and they can start working on the code earlier

although it will be still changing they don't need to acquire as much context and well it's okay we are fine with actually redoing the audit it happens sometimes that we are doing an internal audit we highlight like within a week we highlight some some issues we come back to the development team and the development team was like oh yeah good good thanks thanks we've followed these to ourselves fixed we've we're not looking at the latest latest code fine okay happens we do it again and and again we take more time but that's kind of part of the job and a very important caveat here this is not internal audits are not a replacement to Independent audits the whole point of independent audits is for

the general audience for public to be able to see that somebody else bought the team itself that is financially motivated by this specific project has done the research has put the time and effort into understanding and has put their name behind it because the Auditors especially the most reputable Auditors they're Financial motivation they are incentives are with keeping their reputation clean not with supporting any new rockpool and this is the point here the independent audits have to happen anyway and they have to be done by independent external unaffiliated companies ideally very strong auditing companies

with very strong expertise another thing that we can do is public contests in our experience these contests yield amazing results actually we have received so many reports we have improved the code quality tremendously we received the thing like reports that are not security issues at all just minor qas but this is this all these tiny pieces together contribute to cleaner code base better readability and fewer bugs in the future the incentives in those public contest audits are very different from the traditional audits traditional audits they are paid per hour they are looking code at the code line by line for the

public contest whoever reports the issue will receive part of the Pod and this will be shared with everybody reporting the same thing so it incentivizes reporting unique things unobvious things opaque things things that nobody else was able to discover we've seen people trying crazy things combining different approaches in different places that we probably would not be able to discover easily within the next six to 12 months and then maybe two years later somebody actually would have realized how to combine all of those things together by then we have huge dvl and it's a complete disaster those were really really good this was a really good

experience in my opinion it doesn't replace a traditional audit but combining those is really really good we did two we're planning to do more because we were very happy with the results and finally another thing you can do and that's what we were doing as well brilliant and independent researchers not people working for big auditing companies they're usually much more available they are very skilled if you're nowhere to look and you know where to look because you just go to the top bug Bounty programs and look at the top performers there you go to public auditing contests and you look at the top performers there what we do when we need to audit a bridge we go to somebody who discovered critical vulnerabilities

in L2 Bridges and they can you take a look at Elbridge where you'd like spend a week just look into the code not much code spend the week will just pay you and maybe we'll not be publishing this audit because some of them want to remain anonymous some of them don't really have this kind of a publicly known name behind it but the important thing for us is that we're actually getting additional value of people that will be breaking our stuff later via the bug Bounty and we just are getting a head start here another example for account obstruction implementation the best you can do is go to the public contest on account obstruction cut the top performers and

ask them to look at your implementation Your Design choices so now how this all combines together this is getting more complicated I see in reality is it actually much more messy because the components are not components they are actually split into smaller pieces all of them and they're like interconnected things are happening in parallel and juggling those things and making sure one thing is ready by the other is actually quite hard but when we put this all together at the bottom we have the very same weighting and auditing there's a traditional audits at the top we have the same development process and in the middle we have like internal audit you see it

started before the components are completed and it's completed slightly after we actually do this Head Start due to the spending time and being embedded in the development process we can actually complete the audit within couple of weeks after the the development is done unlike the if we would have started right at um at the completion date on the other hand it takes longer so if we start at the completion date it takes four months otherwise we need to spend five months but we are done in two weeks after the completion pretty good if you can if you look here like component one done done the audit inter external audit is starting after

the internal audit then the green ones are independent researchers looking into it the kind of a blue ish in between our public contest they're all aligned well and they're all happening and Orchestra orchestrated in a way that the internal audit is completed after the component is done and we indeed are sure the code has not changed then we hand it over to the external Auditors then we hand it over to or maybe not complete but at least critical pieces some some parts of it to Independent researchers then we haven't public contest and so on and so forth and only the third component here is not fully covered by all of those things but still our results are our time to Market is 2. like 204 months two

years four months every piece has been covered by independent external audit as it should we had a public contest completed and fix is applied to two out of three components we we actually still have time to do it for the third one for example and independent researchers also additionally audited the most critical pieces of two other two components that we built and now if we go back to the modular advice start with the most critical first which gives you more time to apply all those measures and to do like much more confident in the code that you're rolling out um this is pretty much it of course it's much more complicated it takes a lot of planning it takes a lot of juggling

things around but this is the gist of how we can do everything that we can to protect the systems and also to not to delay unnecessarily the public launch questions well unless there are questions then thanks a lot for your talk or there is a question sorry there is a question sorry um you mentioned you are running public contents contests in parallel to kind of like the official audit firms so how do we handle situations where

during this public contests people find more bugs or issues than the officially hired teams so I must uh probably yeah I cannot go back they were never running in parallel traditional Auditors do not like when somebody else is running a public contest because if this this happens they are not going to look good um so this is why we're usually not doing it in parallel on the other hand for us it's value in any case we got the bugs we reported and we will fix them the more the better it's all right we can come back to the auditing team and say like look how did this happen that we we missed it maybe we should actually

improve somehow the documentation or the way we collaborate but on the other hand if the same things are reported from two different sources or different things they reported from two different sources it's fair enough we have to pay for all of it we like if at the same time to get the same code as in the bug Bounty program and this comes through the bug Bounty as well and it was not a known issue at that point and then we get it from the Auditors later well celery we have to pay and to be to be fair with the community we'll have to basically stick to our commitments and just hope that they help us to discover everything everything that is possible thank you so much

[Applause]