Applying the Pyramid Principle to Ethereum Development / Josef Gattermayer

Transcript

hello we're starting we are now applying the pyramid principle to athenon development by Joseph gatormeyer of a key blockchain security thank you thank you run so you must really dislike vitalik or just for record he's now on the on the main stage so it's kind of challenging having uh having a talk uh in parole but thank you anyway for coming so my name is Joseph and I'm CEO of blockchain we are a security company we are performing security audits and I'm also assistant professor at the

Czech Technical University in Prague basically just what we do besides the security auditing we are also trying to give something back to the ecosystem so we are big Believers of Open Source and we are developing some open source tools and we also trying to get back something to recommodate by uh providing good education so we teach one subject at the University and we are also doing online lessons for for free for anyone we have one School of Solana and then we have school solidity so basically we are operating

on these two chains on Solana and on ebm oops no problem so we are quite lucky that we could be working for some of the top tier players in the ecosystem namely for example our last audit for safe or we are auditing a release of XLR and we also audited quite intensively layer zero and our last audit uh was made for pro bono and because we sometimes even help projects that are building some interesting

things and maybe you know solid a project and it's basically a guy who is rewriting open Zeppelin sort of the libraries to assembly to save some guests so we offered him a free audit and this talk uh has a little bit uh abstract name it's called security pyramid and my motivation for the talk was that uh when a project approaches us basically for the audit in the first day we see with like uh 50 50 to 50 probability if there will be any criticals or not how we do it is basically you can see which security or best practices is the development in

following so for some of you it may be like very obvious because it's the same in web 2 web 3 but I just saw that it's worth mentioning that and if we save one project from making obvious mistakes then it was worth to mention this so basically the torque will focus on anything you can do before the security audit and it looks like this so at the very bottom uh the things have the biggest impact and basically are the most fundamental in the development process and you can see that the audit is like sitting at the top of the pyramid so at the top of the iceberg and of course it has uh it's it's like the final output of the security process but the impact

is uh that limited so basically by having the audit it doesn't mean that it catches all the previous layers and the biggest impact you can make on the lower layers not on the on the top of the iceberg so let's get let's get to the single single layers probably you know already fewer deaths but still it's a surprise that many approaches don't do that so every code should be peer reviewed it's good to follow the standard git flow where basically developers create branches the future branches and the future branches that are merged with pull requests which means like at least second pair of eyes checks the code and this also is kind of like use case

for example a lot of projects we have are written by how I call it like full stack hero so basically like one guy Who develops the back end Who develops the contracts and is doing his own prerequests which uh maybe sounds good for the investors that we have a one strong Dev that can do everything but it doesn't work here's an example it was a solar project but basically uh which thinks it can lead to vendors like one guy doing everything uh basically this is a discussion we had with the developer on slack and the developer basically was not handling some edge cases in the on-chain program and this argument was like I

don't need to handle that because my backend will never send it but come on guy you know we are we are in blockchain you know so everyone can call your contract everyone can do everything with that and this is how it works and this is how it should work so so the the tests and everything should take place in the contract if not of course the Auditors what they do in the audit report this led to three criticals which means basically he has to pay the audit again because with this kind of thing he cannot uh he cannot publish that because of course the community wouldn't accept it as a secular project so that's some of the outcomes of having too much centralization in one person and developing everything

so the next step unit tests probably you've heard about that but still very few projects have them written properly so basically the development team should be writing the tests it's good to uh it's good to test coverage every function for the audit we always say that the discovery should be at least 90 percent uh better is 100. before that of course the Auditors are very expensive developers of tests so it's better to do this internally and also a good practice is if the tests are written in some kind of like different pack than is the tested one or with different person so for example when you sort the

contract it's good to write a test in jscript or python so there's not possible to copy paste just logical mistakes from one uh one linkage to another and also it's important not to test only the happy path scenario but to test also some edge cases that should not happen and of course the test should be part of the cicd pipeline and that's why the selection for example of the testing framework also matters because of the tests take like two minutes so probably then someone will just just skip them in the pipeline so it's necessary that the test execution is very fast I later on speakable some tools that I recommend

so this was kind of obvious and it's the same for web to web3 where we have some kind of specifics is uh static analysis which uh in web free is kind of helpful so these are basically tools that you can run over your contracts and it might find some issues the problem is that uh it has many false positives and uh there are like different tools with different approaches but there's none of the tools that basically catches everything and can replace I know audit or whatsoever even it's like a very sexy story you know we have for the VCC you know we have replaced the Auditors by a

tool that's running on AI whatsoever give me money and also a lot of projects do that but in the end nothing works like this so there's a tool that quite works well but uh what we are always looking for is that the tools have a high Precision which means they are confident about the issues they report and the lower recall which means less noise of the first positive issues uh and if we move to the fourth stage so that's something that just a few projects have and this shows something of like more advanced approach to our security and when we for example open a repo and see that the project is the fastest so

we can say like at the first auditing day okay they probably will pass the audit because just by having this it means that they'll be there are some at some level of the security so fastest again are written by development team and they are trying random inputs on different functions different contracts calling them in different order and with this basically you can be quite sure for example in D5 on mathematics that it covers some kind of undiscovered bet use cases or bad scenarios and it's also good to test the external libraries which is kind of like specifics and what we also do is uh we are doing a lot of

cross chain fast testing and if you will be more interested in this topic there was a there was today a talk by my colleagues called state of python testing Frameworks but just in general fastest single chain is good for example when you are uh main thing and nft so you can be sure that it's limited on Two Chains so basically on chain a is it's it's doing some uh some inputs and it checks the outputs on chain B and there's also quite interesting technique called differential fastest thing we use it for example when there's some kind of like external Library that's not two Battlefield estate as I don't know open the plane but it's uh

kind of like uh not only did but the scope of the library of course X is the scope of the audit so we don't have a time or budget to audit the external library but the library is very important for the project so how you can do it if it's some kind of like math Library so you can take a meth Library implemented in different language that's doing the same that's more established and just try the same random inputs on Library a and Library b and check the outputs and if the outputs are the same it's fine if there's some difference then probably the library B is not behaving as it should and we have done it in one of the Lost

audits so for example the it was a D5 project using the abdk library which is a library for uh floating point uh computations uh and we have just taken its equivalent written in Python it's called Big float and this Bigfoot library is using the C library that's used basically everywhere so we just assume that this Library works and we've compared it with the solid abdk library and we have found that on some specific input it provides different outputs and we have identified that there's like a Precision problem if you compare this to hashes so it's like an on the on the

sixth position from the end so it's kind of like a problem that's not occurring very often but still different output it could mean a problem and the test was quite simple to to test this Library we do differential fastest thing and then is the audit so the audit should be at the very end and for the audit to be successful it's good to provide the Auditors as much information as possible because of course you don't want the Auditors to be spending their time uneffectively so what we always like prefer is that we need developer documentation we need diagrams as small as much as possible

architecture description uh very useful thing is some kind of like video work through the code base and the outcome is that of course when the stages one two four are missing the audit will not save you so the most probable scenario is uh in the audit when we found one or two criticals we just write an exploit for that right in the exploit take some time so for example when we have a audit for five main days we discover critical then we are writing two main days the exploit for that to verify the critical and then basically we spend our time on the audit by writing the exploit so in the executive summary we just write okay we have discovered this critical then we

run out of time which means fix the critical and order the full audit again which of course is not the thing that you want to as a protocol uh here in the audit so basically the audit will just just like hey go and return again so the process still have to do the four stages by the arm and as I touched a little bit of audio topic so there are Audits and Audits and uh at least in our view the audit is not the stamp some protocols prefer that you know buying audit from a high profile auditing brand that just serves good for the retail users as they know the brand

I'm not naming anyone but you probably know uh but the outcome of the audit should be more like this which means some kind of like a very complicated process which security is and uh as a process it's not like single delivery which means it's not like one audit it's always good to order continuously which means all the thing ever release and the audit service not just for the protocol but also for users so it of course should be public it should be very easily readable and by this non-stamp approach of course the audit doesn't tell if the protocol is

safe it just tells that okay we have tried this this and that and based on our findings we see this but the disclaimer is always like we cannot take responsibility but for example uh important part of the audit is something that's called trust model which describes even how for example the treasury of the protocol is handled if there's any multi-signature whatsoever and if we see that it's too centralized and for example allows some kind of like Founders rockpool we always write it in the outlet because the audit is is meant for the public not not that the protocol is paying us you know to give them the stamps so uh recently there was like you know free radicals in a row of audited projects so with a good auditing company

this shouldn't happen because it should be stated in the audit report that this is possible and of course the the audit should hack the protocol it should not like find some low hanging fluid with automatic analysis it should be about the monologue review and uh yeah so it this is how it looks when the critical issue is discovered thank you me Journey and uh with the auditing companies different companies have different values so it's of course good to look for some Auditors that ordered high profile projects and like my personal favorite metrics uh like the number that I look for is how many times has the auditing company been hacked

what does it mean if the company audits a protocol and after the audit uh it gets hacked it's a it's a it's a problem for a Good Thing company on the other side this can happen will happen and happens even to the to the best players so what can what can save the company is the number of projects the company has hacked which means like the many criticals they found so if if for example in our case we uh in the last two years discovered about 62 criticals and we have been zero times hacked which is of course great but even if the number would be one it's still kind of okay on the other side when you discover zero criticals and you are

getting hacked all the time it means that the company probably is worth nothing and the output of the audit is some kind of holy book of course that then the protocol can publish on on the website and I just touched some kind of like best practices here so I will also include uh some specific tools that I can recommend a lot of the tools are written by us so product placement but it's open source joke we are not selling it so for the units that unit tests uh of course there are like two approaches at least so we can write them over sorted contracts even in uh jscript or in Python

we are more for the python world for the jscript testing framework hard hat works pretty well we have developed python testing framework which is called Vogue just disclaimer for the name we found the name first before any movement So my answer is always so I woke we got the python package name first so we just don't care and there's also testing framework in Python called brownie but it's now an abandoned project and there's the same framework called ape and these testing fireworks run against the local def chains so each of them supports animal ganache and hardhead uh as I said important thing with the

testing framework is the speed of execution so we just wanted to focus on this a bit so we have taken the uh 271 unit tests of units for PVP project which are written in hard hat and basically we have forever written these tests for brownie ape and Vogue so this is quite a lot of work and then we compare the speed of execution of this test to it over the three development chains and of course the output what we were hoping for is that work is quite fast here so we can see that this test Suite was executed in three and a half seconds on the annual chain which is for example like 10 times faster than brownie and in our

like view this is very important because for example when we as Auditors help the project to increase the test site then we can just put the test as part of the pipeline which for example was the case of the solid project so basically they accepted the test Suite as a part of their Pipeline and of course if if we will be running one minute they will not accept that so if it runs three seconds it's kind of okay and if you'll be more interested we wrote a comic paper about the evaluation with the sync Frameworks it's now under review but if you just send me message on telegram I can send it to you but we cannot publish that and for the static analysis the most established static analyzer slitter

written by travel bits which has a quite low low uh very very has a very good Precision so it's not reporting too much false positives and then we have a static analyzer in our VOC testing framework which works together with vs code extension that we call tools for solidity so basically if you are using vs code the easiest way how to install bulk is by just installing this extension and it loads all the dependencies and the output is that here is a screenshot of the vs code if you can see see that and basically when a user or the developer for example

creates an here possible or enter and say so it opens a dialog where it's a testing framework notifies the user that there is a possible range so you don't have to run any CI and you can see just like in your Dev environment and for the fastest thing there's a the first and still very good tool from Trail bits called ahidna and we also have a fast testing in invoke where basically on top of the specific things for fast testing we can also support cross chain tests and uh we use it with a lot of projects for example that we are writing on XLR

and yeah as the audit is finished everyone is happy and this is how it should look like so thank you very much for your attention and if there is any question we still have about three minutes I guess not so thank you very much [Music] thank you