Auditors Discussion, Sessions 3 & 4

Transcript

um so I guess we'll do the panel now uh invite all of the panelists to come up here Nick Maddie Michael uh Gonzalo John Anton yep so while everyone gets it I think we'll need a couple more chairs we have enough three yeah I'll take one more here so I have some questions prepared for the panelists I'll definitely ask those and for audience as well I guess what we can do is uh you can send me your questions via email to hi directmodate.blog hi at mudit dot blog and while the panel is

going on I'll try to keep up with the questions and ask anything that comes on my email yeah it's on go make sure let's get started uh I guess we can start with a quick round of introductions I guess most of these speakers have already spoken to you guys might already know about them but I'll ask everyone to just introduce themselves and tell us how they got into this space in the first place hello I'm Neville I'm an academic researcher by training so I was into static program analysis since 2009 and that was punctured by two years working in the gambling industry now the gambling industry is a little more crazy than D5 so you know I'm

really comfortable like in this space so yeah um yeah so program analysis now applying that to to D5 obviously makes a lot of sense so that's how I gravitated here nice so hello okay uh so yeah I'm Michael and uh I started in the space because I uh started mining Bitcoin on one of the school computers and made a little bit of money that way back in 2012 when you could actually do that um and that went down the Deep Rabbit Hole of what his money um et cetera Etc and then eventually uh smart contracts I got tired of Bitcoin for a while because it didn't become the world dominant money in a couple of

years and then ethereum is like oh yes we're going to become the world dominant computer it didn't happen quickly but at least people are a little bit more honest about it in that space and then yeah finally we're here we're okay we need security I my granddad is like 90 and wanted to buy Bitcoin and crypto and I'm like I'm gonna do that for you you're not gonna touch this but you know suffice to say those the you know the young kids the old grandpas basically anyone that doesn't have like a computer science degree would love to use the technology without worrying about rug pulls so that's why I'm here hey guys my name is Gonzalo I'm a hacker since I can remember I think um and uh I'm one of the co-founders of the consensus diligence team we've been going at it for the past six years and

before that people just thought I was trying to hack them for real on the blockchain but yeah [Music] all right uh yeah Medi uh co-founder and director of Sigma Prime um kind of same timeline as you we started in 2016 company was from 2016. uh before that I was a pentester so kind of coming from the web to security space um done you know all sorts of web application security testing networks internal networks external networks for a big four company targeting all sorts of different Industries and what got me into this space was Martin swending who knows Martin swinder here yeah he's a legend he's is we owe him we owe him so

much he basically published a whole bunch of amazing write-ups on his blog very early on um like the first kind of year of ethereum and had found so many bugs on all these smart contracts on all these like internal logic and that blew my mind I remember just like staying up all night just trying to understand what was going on so yeah thanks Martin okay I'm Nick with trailer bits uh I have the world's weirdest resume so this is going to leave a lot of things out but I got into this particular space after about 10 or 12 years an incident response and I got called into an incident response at a crypto company and I saw how that was working and then I got uh brought into another crypto company uh and then with a break I ended

up at paxo says their Chief Information Security Officer um so and that was right before I joined trailer beds so that's how I got here but you know it's a lot more complicated than that hi my name is John I'm head of L2 scaling at uh Quan snap uh I got into this because I was doing my PhD in computer science and working part-time as a baker and then someone at Quantum said hey maybe you should come do something kind of relevant it's still on and I said yeah that sounds way better than baking bread uh so I did that and ever since I've been um just keeping up to date with everything in the space oh yeah hi my name is Anton I'm Alexa funding partner and she's security and uh yeah uh my story is a bit weird as well as usual I was a student and I had

a friend who was also a student but he was more like into economical stuff and he was telling me you know like there's this crypto thing and there is this uh one guy on Twitter who posts a token and I always cannot buy it fast enough like and you're like from computer science background kind of like you do a bot for me and like we did a simple stupid word so as it was just us looking for this coin of a day uh posts on Twitter and trading it uh kind of as fast and this is how I kind of I got the interest of the guy name I think some people may know it so the John McAfee and this is uh this is what's partially how I got into crypto like doing the board

that was trading based on McAfee streets yeah uh super interesting stories uh thanks everyone for being here this is definitely gonna be a super interesting panel a bunch of great talent so the way I'm gonna do it this is um I have a bunch of questions here I'll direct questions to individuals and then I'll open it up for everyone else to add to it or open up a discussion around that question and see where it goes so I'll kick started with a question from mehdi uh tell us like what does auditing code actually mean so I hate that word and I think that's your follow-up question but for someone else so I'll

shut up on that um so security assessments of smart contracts is probably what I would refer to them as um so I can't obviously speak to everyone around but um I mean I would assume that we all go through the code line by line and we make sure that there aren't any known anti-patterns in our the implementation that we're targeting so really it's about understanding the logic consuming hopefully documentation that the projects would have provided hopefully right you have documentation sometimes you know you may need to hop on a quick call with the development team so that you can understand a bit better what their intention is but essentially yeah it's about really

understanding the business logic understanding the potential gaps and misassumptions that may have been introduced by the development team in our case as I mentioned in my talk we always write our own tests always a lot of clients are like no no but I've got my tests I don't want to pay for it no no well it's just good go see someone else we always write our own tests and and that has saved them saved us so many times so really it's a matter of understanding deeply the the system that you're you're auditing you're reviewing but also I think you can only do that if you understand the platform that these things are built on very very deeply so really having a solid

Rock Solid understanding of the evm of ethereum of its security principles of its security guarantees is a prerequisite before being able to provide any type of Assessment Services that's great anyone else wants to add to that yeah I'll add to that that was so he said what I was going to say sort of but but what what I think is really important is this this idea that the clients are saying no use our own tests or or things like that and this comes into a question that's going to come up later which I think is truly important which is why do you want an audit what's behind your reason for wanting an audit I think there's a lot of confusion around what an audit actually brings you and we're going to be talking about that

but it's so important as to what an audit is and it really drives that idea like I can't imagine doing it any other way than what you just said yeah for sure uh the more tests you have the better any value you want day to add yeah so this Sigma Prime accepts clients who have not tested their code in that case it's a good question um I want to say no but at the same time we we'd like we'd like to take clients that are you know potentially various levels of maturity but like they need to understand that the tests that we write are not going to replace their own unit tests right so as long as we have that agreement and like okay oh I'm going to pay for your services and you're going

to be writing tests so I'm not writing uni tests that's the other end of the spectrum uh that that's not gonna work okay cool that's great so um I think we have already established that auditing or code auditing is not a great word for what we do here so uh let's call it security assessments or review yeah or review just saying fair enough yeah that's fair so I guess we have a bunch of folks here uh from different audit companies we can make a decision here uh should we stop calling audits audits uh should we pick a different words Gonzalo uh what are your opinion on that I like got it no I'm sorry uh no we've been through this over and

over again we've all already agreed like multiple times that that audit is not the right word uh We've settled on reviews at diligence um might have like a scattered like audit in communication here and there just because that's what people that's what people look for and understand when like they Google but uh but other than that like we've we've all established that we're not counting nail like the inventory of nails at a story you know like we're like or something like it's it's just not what we do though I must say in I'm not entirely sure if I wanted to fire shots certainly but there are people that do security audits instead of opinions then it's just like

checklist checklist checklist but yeah we we do security reviews you can make can be it from my experience we were be Fortune security sometimes at PWC and only once we did actual security audit it was according to the standard isai 3000 to Singaporean uh government because that's the only paper they can read it was basically really non-informative compared to what Auditors usually develop and usually delivered so it's just something we looked at code and we didn't find anything it sounds like over that lines because actually you cannot release a report where you find something so it's

because this is a kind of a special certification paper sort of and this is the only way that's a fair point so building on that Anton like tell us what is a usual security assessment report looks like what can we find in it uh yeah that's a really good question I think everyone has different aspects in the report uh like compared to other companies here in our reports what we really like to have is like called system overview is this uh kind of our description of what client system is doing what are the parts are there and so on this is actually something we found uh good for us to write anyway just because when you're writing this

documentation you are understanding project better right and sometimes people writing it kind of better than what coin has and they just copy pasting what we wrote to their website as description it's happened but also it's like kind of um parts that we use to kind of really know that we understand everything also like there is a of course finding spars I think that everyone has this there and also we had some kind of executive summary for high-level people to understand what the stuff about because quite often like I people won't be reading the actual

findings it takes some technical skill right so we just do like a section that explains um over the lines that we think the project is okay they're doing the right thing they we have a confidence or over the lines that we did but we believe there are way too many findings we are for sure think there's more and we have little confidence so it makes sense um so does anyone else add anything extra to their audit reports so yeah the centralization section okay now many clients don't like it but it's kind of like uh of course it's not the bulk of the audit but we

also think like what if the keepers like acted maliciously what if the you know the the uh the maintainer of the protocol act technologically and all of that right so we do have a section like that we also tried to adapt like our reports to to the to the customer needs you know like depending on the type of project obviously like I'm guessing like when Sigma Prime this fuzz is like like serialization algorithms like it'll have a different section the same for us like there have been times again it's it's the it's not the the average but like there have been times where we've just had it uh like um agent-based simulations and that'll be like an entire new part of the report so we'll pull it up

yeah I think one other thing that you one other thing that comes up is uh just best practices so like usually there's it's very contentious when you get into opinionated things with clients um around well I don't really think you should probably be doing it that way or you should really be documenting this in the code on the net spec and so sometimes you have to make a differentiation of like what's a security issue and what is just best practice so we say notes and I think other audit firms use a similar approach of just you you should probably do this but we're not going to tell people you're unsafe because of it um but also we would just like to be able to stay you should probably do that um so just trying to politically uh in a politically safe way uh encourage best

practices a little a little bit differently but you know the same obviously we have the findings in the executive summary we try not to be opinionated about um giving any any kind of implication that we are approving or saying that this is good or bad we it's not a seal of approval and and it's not really meant for that what it's really meant to do is let the custom let the customer understand where we think their code is and whether it's doing the things that they think that it's supposed to be doing the way they think that it's supposed to be doing and nothing else the the place where we get into um conversations around how they are doing which really is for internal use and a lot of our customers do choose to

publish this anyway is in something that it's a it's a code maturity Matrix that we where we take a look at various aspects that we consider to be important not necessarily specifically tied to Any Given uh finding but strategically around how what what the findings we have found tell us about the way they are developing their code and those are often contentious too but but they are designed really to serve as a medium to long-term road map of how we can improve overall as an organization not just fix the bugs that we found that makes sense uh the code maturity table is definitely helpful for any One external continuing on dotnic uh when do

you want these developers to actually reach out to you to engage with an audit or even just start talking about it yeah this is kind of how long is a piece of string question um it it it really does depend there there is such a thing as being too early uh and there is such a thing as being too late too late generally speaking is when it's in production um and and people do come to us and say hey this thing's been running for six months can you guys check it out and that tells us quite a bit about the customer um in terms of their maturity and how they're looking at the audit they might be looking at this audit for example as a pen test and which which kind of would go along with hey it's in production is it safe we forgot you know our

compliance guy said that we have to do this uh and and we tend to we'll do it uh if they're if they're doing something that we think is is worth doing but um generally speaking it's when you're ready for feedback uh you don't have to be done but you have to be ready to take in external feedback about your assumptions are are we've made a bunch of assumptions we've set forth a number of principles that we want to accomplish and we would like now we're at a position now where we can slow down and freeze certain part parts of this to take in external feedback and actually make use of it before we get to the point where we're ready to roll this out I think that's that would be kind of The Sweet Spot Michael you wanted to add to it no one oh wait yeah I'll I'll throw in

just like yeah I mean obviously there's going to be backlogs that you want to get ahead of um but the other challenge I think is knowing what the ex what to expect coming up to the audit um especially as things change like if they come to us uh six months ahead of time the code base might not exist so estimating okay so what is this going to take can be can be more difficult which is a very challenging area that we're still dealing with but I think the other thing is um okay what are you going to be looking for like what is the level of documentation that is going to be required what is the level of testing we would expect and we've we put out like you know education like this is what you should do before an audit some clients follow it some clients don't

um we try to kind of like carefully hint that if it takes us longer to read your code because it's badly documented you are going to have to we're gonna have to charge more time um sometimes that encourages people sometimes it doesn't but I think even if it's just basic just hey this is like the checklist of uh what you should really have ready before you come to us for an audit and if you're coming earlier than that and then either we can still do work with clients that can be like um we call it security you before an audit but maybe there's a better word we can all agree on but basically like before you even do an audit we can give you much more granular kind of just basic feedback before we have to get adversarial about this about to go like this is about to go into production

maybe earlier in the cycle how should you architect this how should be building it out before you get dinged in an audit and then have to spend all this time refactoring so it's still a life cycle that changes with every client but like earlier is better if just for that you know I forgot to say one thing and that's a really I agree with what you just said one of the things if you're thinking about getting a Security review getting an audit it's important to think about it as early as you can because there ain't none of us up here who's got like time on Thursday there's a there's a long waiting list and and that's because there's a lot of people who want to get their stuff looked at it's a really great uh way to look at this waiting time that once you've got the

appointment and you've got it all set up look at that is the time to get prepared because it's going to take a while and and there's a lot of good stuff that can come out of that waiting period it's not just all awful because we can't do anything we really encourage I think all of us up here would really encourage that and I'm sure every one of us has you know we have office hours we have like sort of free things for people who are waiting for an audit so that they can prepare as best they can so when they get to us they're going to get the best value yeah that makes sense while talking about audit preparedness Michael tell us some things uh teams should do before reaching out to you uh so we have a um by the way is there two Michaels here

no okay just making sure um there's sometimes two Michaels um so basically uh there's something that specifically has been worked on that actually just came out a couple weeks ago um there are some people on this panel that have actually contributed or from orgs that have contributed which is uh called ethtrust it's something that was put together by the ethereum Enterprise Alliance and I think there are other resources that people should go to but it kind of does the best of like summarizing all the things that are likely going to be checked for in an audit to kind of give clients the heads up of like okay before you go through the Maelstrom this is really what you should be looking for um and I think most helpfully it tries to divide it into stages there's the first stage of these are all really basic things you could probably catch in

a static analysis tool like hey you're using transaction origin uh please don't use it to verify someone's sending you money um and then you can actually create overriding requirements of like okay you're using transaction origin but maybe there's some weird reason you think it's useful um okay the next phase of that after we've done static analysis is the manual review we now see are you actually using it safely or are you encouraging any known issues and then maybe even going to the next stage is the quality review okay we're using it but are you documenting it are you telling people why are you using it are you is it all in the nap specs and so like are you really ready for production um so by dividing it up it's a little bit more approachable for people that might not be at the audit stage

um but it also like I think the tricky thing with standards and this is a first pass on the standard so everyone please give feedback uh uh and gripes but basically it's not trying to say you can't use it it's basically saying okay these are things that we're generally considering safe and there's like a Tran there's basically overriding requirements that go all the way down to the the final step of the quality review of like okay in the end auditor please check this is actually safe um so there's at least like a good mental map you can go down before you reach the edge of the map of we don't know what the future holds or we just there's so many possible ways to do like re-entrancy that we're not sure but at least you've tried all the basic stuff to make sure that an auditor doesn't

have to go to the trouble of like checking it themselves yeah sorry if I if I may there's a lot of great tooling out there please use it please use it there's a bunch of people here A bunch of companies here that's been working really really hard over the past few years mithril Slither a whole bunch of others please use that tooling it's it's going to save our time it's going to allow us to focus on the business logic type flows the really meaningful bugs that are probably very hard for you to find but there's no excuse nowadays not to run Slither not to run me through before seeking a security assessment and I'll see another thing so like I totally agree like diligence was involved with with these stress levels and all of that and like

standardizing things is always great but like out again like a note of advice which is like leveling it out may may make it seem like some things are more important than others but then again I'll just repeat what I said in the talk like nothing is truly better you know like just more things are better so so just like categorizing them it just like serves its purpose but like don't be fooled by by the leveling system things to lay your teeth right that that's uh yeah I love that so I think I want to jump in here quickly um the tooling is great and definitely if you've got the resources in time 100 do this but we you know we occasionally still see people come to us with no tests and no documentation and

it's like the tooling is good but if you don't know what you're building you don't have a spec you don't have tests the tooling's not going to really help really help you because for all you know you want that specific error to be there and you know we're gonna tell you it shouldn't be there but yeah there's so much to do before the the sort of advanced stuff doesn't have to be hard just just put the work in you know make sense um so John tell us like what should the end users uh and declines expect from a Security review yeah so this thing this has already been said um feedback uh so I think feedback most of the time on another report will be a bug uh I don't think there's there's been too many bugs or reports that we ship that have been totally bug free

um some some issues are sort of not bugs and actually features and we don't necessarily see that and sometimes that's because uh the documentation is not clear or we haven't sort of fished that out of you um from the communication we've had but feedback in terms of tests and what's actually the code is doing just in general is is going to be um there right ideally it's good feedback that this is mostly good and you don't have major major concerns or it's big concerns and when look here it is please fix it before you ship but I think the more important thing is actually what not to expect which is a strict guarantee that this is this is going to work everyone here I don't think anyone guarantees it here and even if you do formal verification right if

you if you don't prove the right invariant that's it it's over right like or if you didn't think of an invariant and it was super important um after the heck well too bad it's it's too too little too late so you know feedback is is the positive thing you get because you do get to improve the quality of your code and project every time I think uh like I said I don't think we've ever shipped anything that's empty uh and the thing to not expect is that this is a stamp that says this is run perfectly forever it's hopefully it does we we really hope it does but you know it's not a guaranteed yeah definitely makes sense um so I'll put in a few thoughts of mine uh from my perspective like security is not just about final Security reviews I think

it's a continuous process so I do want security people to be involved from a very early stage when there is no code uh even when there's just a design document let's say when there's just an idea till uh and be involved even after it's deployed even after it's in production uh you keep discovering new attack vectors new techniques uh the protocol layer itself keeps changing which might make something vulnerable now so I think a continuous process is definitely nice uh with that respect uh we have another question I'm taking some audience questions now uh so I'll say like which of the audit firms have in the past have you guys audited

or done Security reviews on non-code things like design documents specifications and stuff like that well we do um we cannot mention the client but sometimes it does happen but we want to can say is that you know typically it's it's expensive to be doing this like some of the projects that are starting out um you know they can only afford to have one audit and you know they make it count by you know like uh making sure that it's done exactly before it's deployed but yeah we we do that sometimes yeah we're starting to look at this from you know once you hit an audit Point you're usually going to production into that point it's kind of like

um Good Luck see ya um and and even though there have definitely been attempts to pull us into the operational role of like okay you be you sit here with us and help us press the button we're like well we don't do that either but what we are trying to do is there's been a lot of good um uh let's say comparisons to avionics and how like the avonics industry has like helped to mature code before it goes into production it's in a plane but there's also still going to be a need for if you're in the plane and you're flying it and then a red button turns on that means something has happened is okay then what do you do and I think one thing we're trying to do more at open supplement is to help Define like what is the checklist of as Auditors we have

a great understanding of the code there are ways that as a follow-up to an audit we can also help to figure out how they can do incident response and threat modeling from a training perspective or process perspective um but still give it up to the the teams of the communities are going to have to figure that out so um you know once even once you hope the code is safe there are still things that go out of sight of the assumptions oracles turn off protocols upgrade that you weren't expecting and so figuring out how to respond to that is going to be the next step I mean we we've also done that in the past but to be honest like in and I I even did some some of those things to be honest those were um one of the like most horrible experiences when dealing it's I gotta

say the reason for that is that like so this happened if it's a retainer right like if it's someone that's not in the retainer it's just a continuous thing and you're already used to the to to to to the client or like to the people that you're working with and you like them also like that's very very important but uh it works better but we've had cases where I was just like reviewing the design phase of some projects with no prior contact and it's just a miserable experience because like people are not ready to hear and and like they're if if they're not big-minded they're just like really really strongly just like glued to that one concept that you tell them dude that's not gonna work you know yeah we do design reviews as well but mostly

for our long-term clients the retainer type type that you mentioned it's also not very intuitive if like if you're like budget constrained unfortunately these days so some of our clients that I put up that are not and they get us to review what they're thinking of building uh which is cool which is very interesting and I I mentioned that we love fuzzing and I'm very happy to do that when they have an executable spec that's that's the dream right like they've really thought this through they have an executable spec that I can fuzz so not only I can provide feedback on the overall design and the various pieces of architecture composing their new protocol but I can also actually tell them where it might actually go wrong once it's implementation time

so we find most of the issues being economic attacks do you guys also have the same experience you know when when you're doing a design audit so that's interesting we don't um we don't claim to be performing economic reviews that's that's uh that's a that's a very important thing that we clarify with our clients we make sure that in implementation is matching whatever spec or whatever design has been uh put out but obviously in a lot of cases you can easily spot things that go wrong and particularly with fuzzing fuzzing is great at finding those edge cases that if you're just staring at this mathematical equation for hours and hours you wouldn't be able to find

yeah the all the same things echoed um we also we also helped with with formal threat modeling which I think is actually really important to to get that down on paper and really talk to people about what how their how their things look to to attackers and give them a sense which brings me to this other point which is when you are getting an audit this relationship is really intimate it's it we're going to know as much about the code as people inside your organization warts and all it's a really really close relationship and the the part of it that requires maturity I think on both sides is that we have to be willing to look at what it is and we we understand what we want it to be we have to be able to look at what it is and it's not contentious

it's constructive and that I think if there's one single thing that that sort of trips us up when when clients get frustrated it's if they think that we're insulting them or think that we're saying that this isn't good it's not about that it really is around hey we understand what you want it to do you know that's over here you're over here let's help you get there and so all of these services that we're talking about all these different ways of reviewing it or ways to help customers get there incrementally yeah I mean I guess we're still on topic of like design reviews um we've done it too I think I think we really like the this phase where you have a good spec to the point where it is executable and ideally you can run it throughout the improver or something like this um I don't know if we've done tons of it

but I think we have done it I mean I think it'd be great if everyone could do it I just don't know that everyone has the resources for it because like clients it drives like price up for the clients and it it takes up our time so when you do the economic reviews do you model things out or is it like uh purely human brain well most of it is uh human brain I mean we've been involved doing you know economic attacks like you know and doing bug bounties and that sort of thing and some some of them were economic um yeah I mean we do kind of like have someone who is a you know PhD in math he he can model um derivatives and things like that so that guy would get involved in something

like that too yeah I asked this because like I know that even though it wasn't for a client I know that for example you aren't um once modeled something like uh I probably can say who but like muddled something a system in cat God but it's a very cumbersome process that's why I'm asking it and definitely like like like Maddie said like it falls a little bit out of what we think is our our job and sometimes it's obvious you know someone comes to us and say I'm gonna create a stable gun it's going to have this mechanism to you know keep the back which one which one [Laughter] no we didn't work for them for sure so I guess the t uh key

takeaway here is that everyone does uh like sick design document reviews and works uh throughout the start not just Security reviews at the end so uh I guess my suggestion to everyone here is like if you're building a product get these guys involved throughout the whole process not just a Security review at the end but and be like accept feedback as Gonzalo said if you be like if you are very strict about your specs and your ideas you're not gonna get the best out of these people so uh like uh be adaptable change accept suggestions and be better so this question was from Thomas from the audience thanks for it his second

question or comment was uh since the audit queues are so long at all of these companies he's wondering why don't you guys raise your rates I'm not gonna ask that question because I don't want to give you the any ideas moving on so we have a question from Kurt uh directed towards Neville so tell us how does Watchdog recognize an execution Behavior as a bug okay so uh first of all recognizing many different execution behaviors out of just code that's you know that's difficult we use static program analysis to do that now for specific executions that are you know

um particular we actually do write um you know vulnerability bug patterns uh based on the abstract execution flow of a smart contract so I can give you an example let's say for instance you've got a transfer from okay the second argument of transfer from is an important argument and it must not be tainted by some user that does not have owner access or anything like that right so hey okay so so we use for instance taint analysis and detained analysis is um you know it also takes into consideration the symbolic

conditions that need to be satisfied in order to reach for instance that transfer from which is special not us cannot be tainted so we've got you know domain and also you know got you know classic program analysis and also symbolic all of that combined if that makes sense uh moving on to the next question can I actually answer the point about the prices uh okay because I actually think it's important like oh all right we just I I was just quoted in the in in something last week saying that we'd raised our rates about 30 in the last few months and I think that there's a lot of pressure because of these extreme waiting times to to raise the rates because our our actual costs have gone

up and also there's a lot of people trying to get in who are like me too VC funded crypto Bros who are just like doing something that they're like here look at this thing and and we we want to be discouraging to that but we also and I think this is the limiting factor we want to encourage people to do the right thing and so we can't just arbitrarily raise rates still there so eye-watering people go away it's a really difficult balance to uh to keep out their Tire kicking kivitzers and and let people come in who uh who who really need it and who are doing really interesting Innovative things I I don't know if that's the same for everybody but that's what we're experiencing it's a really tough balance yeah that's uh definitely very nice answer

so moving on to the next question from uh s beta apologies for the pronunciation obviously uh so they want to ask like in the security reviews that you guys do do you also consider nation states as possible attackers or sanctions they can put in or laws they can or regulations they can put on I I guess it there's a lot of different ways that could be approached but I guess the biggest one is the the governance one obviously so there's assumptions of these actors will act honestly and behave and behave in the interest of the protocol so I did some fun like side research outside of my day job to to work on an article for coin Center that was based on tradocation like I said this in the talk but you know they built that system to

essentially not be controlled by anyone even the devs or the Dow themselves so we don't evaluate for that we don't specifically say oh the Government tried to shut this down you're safe or you're you're not but it's inherent in the assumptions that we make about who does have permissions in the system and what can they do and if I guess if you're reading a report and you care about that you can look at the well okay where where are the majority of multi-sig holders uh living today assuming that they are they're not Anonymous um that's about as good as you could possibly get and then when it comes down to uh ethereum level censorship then all bets are off yeah like that that feels like like if we're talking about worrying with nation states like at all layers in the stack then that's impossible you know like

it's like if they shut down all power in the world then yeah there are no computers you know like I mean the attack surface is the same for everyone right like you're a hacker when you grandma's basement you're sponsored by North Korea like you're dealing with the same entry points right so I'm not too sure I understand the question um too well to be quite Frank yeah on change there's kind of no special government addresses or is it the each address can do it mostly the same except like some privileged roles and contracts so yeah I mean the only you could say that government can have better access to the capital or like do some manipulation attacks that's based on

validators right but this is kind of I would say in most cases are out of scope because yeah who knows what they are capable okay wait no I gotta do I gotta do a quick call didn't trailer bits do a great and I read a blockchain article about uh centralities in crypto and they they did go down the stack oh yeah this is good yes we did uh and and I think that many people thought that we were poo pooing blockchain technology in general actually what we were saying is there are risks that a lot of people are hand waving around and and we're not you know we we're a blockchain really blockchain Central company we think that blockchain is fantastic and Innovative what we want to do is articulate risk in a way that people can understand it so that they can take risk mitigations and

I think that after the initial brouhaha over that came out and we we actually when we released it we we intentionally didn't talk to the sort of cryptocurrency press because people were just going indiscriminately nuts but we did do a lot of time talking to Regulators we did a lot of time talking to to large institutions uh who who are looking to understand the risks because I think if there's one thing that we can all agree with is we're in a fast-moving industry this is fast-moving technology and things are happening um in parallel so there's a lot of things going on and if if we're not good at one thing as an industry it's describing risk in a way that is

actionable we're kind of just like well if this is a good thing it's new it's great let's go and we just want people to slow down a little bit and take some time so thanks for bringing that up because it's a cool paper it's called unintended centralities in blockchain and it's it's on our website I liked it I I remember we got called out as opened up in the library has a lot of code and I didn't actually know what the takeaway was but I was like hey we got mentioned yeah I don't know if being a centrality is a uh is a a good thing or a bad thing but you know well yes foreign if anyone has not read that report I would highly recommend it's a super interesting read so moving on right now we know like most of the web 3 code is not very mature

compared to what we have on the traditional side so Peter wants to know what needs to happen in your opinion uh for web 3 code basis to be more mature do we need like more standards do we need something like ovasp or what should happen I think I can start in my opinion I partially in my Talk touch the standardization problem exits interfaces that we have are like probably not the best because they're not really describing how you should actually integrate anything I guess we are a good thing we'll be creating some common formalized standards that actually describes not only what four byte

signatures you should kind of pass and to trigger the function right now but also describes is there like re-entrances inside this contract right is there some dependencies on external like protocols like uni swap right what they're using is price heat right and it should be not uh pages of documentation that you need to crawl and but I don't know it's some kind of I think I have uh in I had a standard where it's all stored in some kind of database and you just kind of can uh if you want to integrate new token just go there and see what are the trust assumptions and what are their Integrations

yeah I mean like I don't think standards are going to save us like I think as everything matures like the tooling um everything around the around the ecosystem as as it matures so will the code and so does the code right now like if I would I would be scared to look at solidity code from you know like 2017. [ __ ] no my God yeah I I know uh one of the previous talks I think mentioned traceability um and that was like okay you you start in the very beginning of your idea maybe you did threat modeling maybe you didn't but maybe there's a spec there's a yeah there's a Improvement proposal there's a white paper whatever and eventually you build it you uh test it you do a lot of things and eventually you get to that

audit stage and hopefully you've done all the things that we would all ideally like you to do um but we don't actually know if you've done them unless there's traceability into that pathway and then even after you've done it and you've released it and here's like a pain point we I think a lot of us have is like when they deploy the code and no one needs to have most of the time people don't actually know if that code that they deployed was the same thing that they mentioned in the report um or even if it's been upgraded later uh and then after that how is it being operated who has control of it like there's so many areas that maybe to us like I can look at the etherscan Block Explorer and figure it out going through the code but the vast majority of people

have no idea what that is there is a and I think that's how like phishing attacks and a lot of other things happen but also just general misconceptions about what is secure and what's not so if we have that in like a plain language or at least a plain like timeline that would first of all would help with audits because then we can see what you've actually done up till now and then after that it would be helpful for everyone else to know you've been audited and what you're going to do after that it makes sense um so in general in panels I like it when there is some like disagreement between the panelists it's more fun so one thing I'll touch back on gonzalu mentioned I think that code bases that are split in multiple pieces are hard to read which

are like very modular uh on the other hand open J plane it's probably the most modular code base out there so [Laughter] it sucks wait let me Trace back let me Trace back it sucks for us you know but it really helps the developers and then the the open sampling libraries are structured that way because it works you know it wouldn't be otherwise it it actually works and it needs to be and I'm okay mind you I'm more okay with reusable thing reusable pieces that are consumed all of the time by the developing teams to be structured that way uh but not

with the entire code base being structured that way okay like the I because like these components they're they're really well known we have this module in like our vs code extension that is called like the the ethereum doppelganger or something like that and we already have the the addresses and like the the GitHub URLs for like all the Oz libraries and their versions loaded up and we just do the doppelganger check so that's easy for us but like that usually that is a okay there's like a separate part of the code base I'm okay with that but that when people do that for the entire code with such an overload in in the best the but the best audit that I've ever done myself I think it was like the the RICO audit for for look so

Fabian Vogel Stellar did the entire system in one file you know it's like start to bottom comments and like lines dividing with like multi-line cons I love that you know like it just reads better it just feels more natural I don't know I can also say this is something wiper does by default like there's no inheritance and um also speaking about this inheritance in solidity I also not huge fan for it because if you like try to have a contract that inherited inherits multiple countering and you need to find what actual open sipping function like default truck and transferred us I mean it's really

understandable pattern and nobody opens happening but sometimes people organize this way this function is redefined in five different ways right and uh you need to basically look at an inheritance tree to actually know what is triggered yeah I mean we actually internally have like actually like a plugin for vs code that kind of you compile the code in the like Foundry and then you're relying on the ASD of uh of the compiler like you know puts you exactly which Target you are calling there so we are relying on that to kind of Jump Around the code instead of trying to find which functions like by buggers if you just do a search based on a you can have like five different functions you don't know

which one is it sometimes what we do is we compile them and then we decompile them and and then the decompile representation gives you that explicitly right it doesn't work too I I appreciate and agree with what was basically everyone said because like the thing to think about the open sub library is that we're building it for like almost anyone to use potentially for anything so we're like trying to cover a lot of bases with modularity it doesn't mean you have to write your code base to be modular if you're doing it for your one thing and you're not like pitching it as oh but everyone should inherit this uh and do it in their your own special way like you don't need modularity we built it because it's flexible for a wide array of use cases if you're doing one use case just build it for that use case and

you don't have to over architect it because then yeah it'll probably be a cheaper and easier audit for all of us I'm gonna this was too agreeable we need do we need space here spicier questions audience help me um anyway so I'll pass on my comments on this as well so as both uh security person and a developer modularity is great when you have reusability of code base so you should have modular code base if you are eliminating code duplicacy but if you are not using that uh code anywhere else then modularity doesn't really help much uh then you can go with a monolithic architecture uh within the monolithic architecture you

can still follow the principles of like one function one like task and stuff like that but if the splitting of stuff between different classes and different contracts and all of that is only really beneficial when you reuse that in a different place both from Securities point of view it's good to do that and from just a code hygiene perspective it's good so uh I'll take a break from the audience questions for a bit and return back to my questions if I can find them so Nick one question for you uh should be interesting I don't think we'll have disagreement on this but a lot of the community always

says if a protocol gets hacked which was reviewed or audited by one of the Auditors they should be liable what are your thoughts on that um okay it gets us into the the point that TT was raising a little bit earlier about testing Auditors and making sure that Auditors are finding things um let's talk about expectations of an audit my short answer is no but uh the long answer is uh it's no because there are a lot of factors that go in to um whether an auditor or an audit will find we're never going to find 100 of the bugs unless we have unlimited amounts of time we never have unlimited

amounts of time for a number of reasons that are beyond our control and usually those are your budget your production schedule your sales promises to your customers and your ability to get into our cycle so basically most of the time people will be like yeah we want three weeks and two Engineers so we're we will find 100 of the bugs that can be found by two engineers in three weeks and that is no guarantee and I think that this has already been said um the the idea a little bit earlier of um hey let's let's insert bugs in so that we can see if our auditor is doing a good job well it helps me because y'all are paying by the hour but I don't

think it helps anybody else I really don't think that I mean if we find all 10 of your bugs but none of the bugs that you didn't know about that you put into your code that's not a good outcome for you and that I have a serious problem with that as as somebody who runs an audit company I'm not giving you value for your money because I'm only finding the little things that you put in by the way it was also said that those bugs that they put in they went out of their way to make them like stupid right because they didn't want to put in any foundational things that would make them do too much work so now we're finding stupid bugs that you put in as opposed to stupid like maybe non-stepit bugs that you didn't know you put in so that's where I am on that and

no I'll I'll add to that and I guess I'll say the the liability problem is down to and again we're because we call them audits but in reality they're security assessments there's an assessment or there's findings but there's not a there's not a part where you get to the conclusion and it's like you've passed or failed right and clients don't want to pass or fail because they might fail um so in that sense and sometimes we we try to strongly word uh you know oh you know by the way there was a lot of bugs in this we only had two weeks it might there's prob I mean we never give a guarantee but in this case there almost certainly might be more um and those are the scenarios where you know you don't want to bash client too much in the report you want it to be

cooperative and and uh constructive but you want to be factual right it won't be factual yeah yeah and so in the end it's like that's that's where the liability lies it's like it's what you do with the information I mean we give that information you and you can just decide not to deploy it the only live the only liability that comes in not just like legally but ethically is did we give you all the information we had in the most direct and clear Palace of a way and make sure you understood it and and have the follow-up conversations of just like sitting down with you and saying yeah okay you understand this line by line you understand why we feel about this we might disagree but at least we've made the case as strong as we can and no I just want to say that it's very

important for everyone to understand that these aren't audits these are time boxed security assessments or reviews time boxed time boxed yeah you if you give me more time chances are I will find more bugs right I had a client recently gave us a bunch of time very complex protocol gave us maybe I can't remember like eight personal weeks off the top of my head the report came out with oh my God like tens like I can't remember like 15 crits 20 highs um and obviously you know they they got a fair bit of value for what they paid for and at the end of the review process the client comes to me and said so we're good right we're good to launch so yeah it was it was a difficult

conversation because they did you know end up paying a fair bit of money for our services but unfortunately the the answer is not yeah yeah no go for it you're you're sweet no no hell no hell no you should probably be hiring other Auditors to take a look at the code base you should probably be doing more internal testing you should probably step back and try to think about why there are so many critical bugs in your code base but unfortunately the state of our Industries are such that a lot of people see our services as the Silver Bullet I'm going to be you know coding and coding day and night but that's fine trail of bits got me covered consensus diligence got me covered right and then once they're done with their work no

matter how many issues have been found will be sweet we should be able to launch so yeah campaigning hard to change the mentality across the the ecosystem but you know it's difficult I think that I think that like like obviously not liabilities like Auditors can't build it like we're not insurance companies you know like it's like we can be but we charge a lot more one of us might be but that's another thing um the the point like I just wanted to add that like even though like I fully fully disagree with like uh Auditors being liable for for hack side I do then that's the wrong way to go about it like

I think that's the entire wrong way to go about like creating more alignment so I'm not okay with liability but I'm okay with more alignment like I think like there's no harm no falling like in just having more alignment with the client whether it be like through token payments not not that we have ever done that for example um but uh but uh but I think those things are fine to experiment with but not liability just alignment different things yeah to to that end we so we if a customer of ours gets hacked um obviously we get involved uh and we will help with their incident response by helping them by by looking at the area uh first of all to see you know did we miss anything but then when we find

out that we didn't uh then it's really like okay we we did look around that area we can give you some assistance and we can help and and we're never going to leave them out on a limb I guess everybody here is doing exactly the same thing we that's the alignment that we're talking about because now as I said we know your code almost as well as you do so that means that we are our really critical extra sets of hands at a time of incident I'll just add add one more caveat that I think is also shared just in case anyone misunderstood it it's not we are not your incident response we will assist you Friday eight to five et cetera maybe and this isn't to say we won't try to help as soon as we can but there there are limitations to that

please have your own ways of doing it thank you for saving me for myself yes yeah that uh yeah that's all that all makes sense uh people need to understand that Security reviews do not guarantee anything uh they are just as maybe say time boxed Security reviews you say that again time boxed Security reviews yeah that's your key takeaway from this so we are almost at the end it was a very fun panel I'll uh end the panel with one more question trying to split the panel uh I'm not sure if I'll be successful but upgradability feature or bug let's go around the clock I'll be blessed

right so yeah I mean I I agree with upgradability of course there's been a lot of uh you know uninitialized proxies and all of that and you know we found a couple in the wild uh there's still another one which you know we don't know who owns it but you know there's a few funds locked in there um yeah I think it's a feature but it needs to be implemented in a way which uh Which is less modular I guess um so yeah yeah I mean obviously I'm going to say feature but I'll say that uh it is a feature but it also enables uh bugs of people in governance so uh handle accordingly I really do that's fine that's fine I'll

get it um I think I really liked what Michael said in his talk about the fact that it really really depends on what you're building um recently I had I was reviewing a report written by one of my team members and they raised they wanted to raise a high severity finding for a token contract being upgradable I'm like I get I get where you're coming from auditor like I I feel you but it it's not we we're not here to judge we're not here to make judgments on what people want to build right as long as things are clearly documented as long as users know that that token contract is upgradable who are we to say that they shouldn't be

upgrading um so I don't know it's a tricky one and I again I really like what Michael said earlier oh I like that I'm gonna I'm gonna go with qualified feature I'm going to say this is a religious debate um and I don't really want to get pulled down into it but I will say that if you are going to do that start planning early start planning at the very beginning when you're planning the very very foundational thoughts and make sure that you follow the steps very very carefully because while it can be a feature it can also bite you in the bum unless you are very very determined and careful at every stage to make sure that you're doing things correctly I'll also say a feature but with great power comes great responsibility I think

there's there's a lot that can go wrong and I think honestly I think we've already touched on how you can butcher your own contracts by upgrading them wrongly or or at the wrong time I think what people actually miss the most is that they interact with something that can be upgradable and they forget that so I think that part might be more of a bug and I wish there was some way to sort of lock in a version better in a consistent manner but I think a general feature oh yeah it's features that needs to be carefully handled and I mean I think it's kind of just a modern matter how to do these things and I there is a lot of Alternatives but I I'm not sure like it's developers are educated to know how it can be done other ways right so it's for example if you have a

protocol that is integrated with 10 other tokens and all the stockings are also upgradable I mean you need some sort of availability yourself right you cannot I'm the option like against our mentions you can redeploy the contract every time one of your depends has changed right but there are other Alternatives like doing adapters that you can redeploy every time right and this is uh kind of another way how to you can handle the availability of your dependencies right I I guess there is I think it's definitely overused feature in my opinion there's better ways it's just people don't know them that's exactly that's exactly that's what I was gonna say I don't hate upgradability like it's a thing maybe I do yes yeah I do I do I do

but even still like it's a concept like I don't think I could hate the concept of something being upgradable you know like it's hard I just hate the tools that we're using today to do it like most of it is bloated it's not needed like I hate how little people think about it it's like oh for good measure I'll just throw in some upgradability I might need it in the future no you won't what are you talking about you know so I don't hate the concept I I definitely think that upgradability is a feature I just think it's a really annoying one like really annoying one yeah definitely that makes sense so couldn't split the panel this time but we'll get them next time uh anyway it was one last question might just please who has the best logo

thank you thank you I've heard Sigma Prime multiple times since is it Sigma Prime okay I think I think you won this one okay thanks man appreciate it thank you okay I like the logo I want to give it like change security like runner-up just because like I like fractals thank you we don't spend money on frivolity everything yeah bunch of interesting answers and it has been a really fun panel uh thanks for being here it was nice talking to you all uh we'll take a quick break here 15 or 20 minutes you can have coffee outside thank you thanks guys thank you