Mutation Testing for Smart Contracts

Transcript

you your life you can start all right so hello guys so my name is Johanna I'm security researcher at ethics and well

as you can see observe a talk today about mutation testing but first I'll give you a little bit of context about what we can do ethics and based the new effects platform which allows you to have an easy interface to find bugs and common vulnerabilities in your smart contracts the second thing that method can do is look for violations to properties which is specify using assertions of some sort and these two aspects contribute in large part to secure smart contract development which

is the topic that we care a lot about and last year I wrote a research paper and released the tool called vertical which implements the testing technique of mutation testing and this is what I'll be talking about today and I'd like to start out with a question but unfortunately our feedback is a little bit difficult so I'm just going to assume that whatever the answer is to this question so do you use code average and the answer is probably yes I'll give that a few big open source projects and and as you can see they

report code coverage and some of them actually pretty high code coverage percentages 99 percent 98 percent and this is sounds really good does it actually mean anything does his high score mean that you now trust your test and well this is maybe a trick question because and and I will show you why so here I took some units us from the magical in truffle box and remove some statements and if you look carefully you see that some of the variables are

grayed out which is into whatever moon and Beyond code coverage so the code coverage between these unit tests and these is exactly the same but the second unit test and do you actually test your code they check for cases or the initial ones where I remove the assertions don't actually do any now this is maybe a little bit of an extreme example but it's meant to illustrate like the importance of the actual assertions and

values he used to assert that it has actually succeeded one thing that might have happened to you it certainly happened to me sometime is that I had an assertion where compared be expected and the expected value instead of the expected and actual value so this would actually be the same as if I had not written the assertion at all and which is bad and of course this leads to the question shouldn't I use code coverage anymore and the answer is No shoot still use card coverage because

code coverage is very useful in bed it tells you what you certainly don't that and what's awesome about bed is that you can take your code coverage and you know immediately how you can improve your test suite by looking at the parts of the code that you don't cover mean you like to test for them the only issue is that god coverage doesn't really tell you anything about the code that you do cover so you want would maybe want some more information about that and that's exactly what mutation testing tries to achieve and as you can see on the slide and mutation testing tells you exactly how efficiently your tests detect bugs

and we'll go into how mutation testing try sensor to give you this information and the point is that if you know how efficient your country is at finding bugs specific in different areas in the code and you can also know where you need to improve your tests to make sure that you your your to speed increases in quality and that you're able to capture more bugs or written more about our from from being in your code so the general instruction behind mutation testing looks like this we start by generating a set of bugs and we do this using a

mutation operators we like mutation rules that specifically try to introduce certain kind of bugs in your code we end up with an entire set of mutated programs with all these different kinds of bugs in them the next step we'll take the test feed you have run them on each version of the probe each with a different bug and see if the test suite succeeds or fails it'll test seed 360 it doesn't really detect the bug so we say in this case the bug survived it's a discrete fails it does actually find the bug and it

helps you to know like something is wrong when we say that the bug is kill and finally once we have all this information we can find the specific areas in the code that are not yet tested as far as liken it to me and then improve those tests or add tests to get this increased level of quality and I'll go more into the details of a specific phase so first the first phase generating bugs is actually called generating mutants so when we have the original program and we change it to

create a program with the blocking it we apply a mutation and this mutated program is then called a mutant and how we applied this mutation is done according to mutation operators which is basically a fancy name for like a mutation strategy or set of education rules so in the tool I implemented and we first looked at existing mutation testing tools specifically P I T which is a mutation testing pure Java and we looked at the mutation operators they enabled by default we specifically looked at those because they have done

some analysis where they found as these mutation operators were most effective and I didn't trade a lot of so-called equivalent mutations which we'll get back to a later on in the presentation so there's a whole bunch of mutation operators listed there and in the next slide I will give an example of them we also increment into two new mutation operators to mimic but in vulnerabilities the developer by accident and I'll give a brief example for that as well

so here we have some examples of the mutation operating operator mutation rules the arithmetic inversion changes like additional factors into subtraction operators and it's mostly pretty straightforward I personally really like the removable mutation operator which basically looks at function see if they have a modifier and then which permanently removes them to create this mutant but what this will create or effectively test is it will see whether your test could actually test for

permission control and authentication because the these modifiers are usually used for the me before this this kind of logic like be on the owner/operator as you can see on the slide here and in doing so the mutation testing will not only tell you like whether on these comes about are being taught but also where in the code you're not really looking at for these specific kinds of bugs which can be really helpful so the next second step will be the mutant evaluation and as I previously mentioned

you will run the test we can end up with an alive mutant or a dead mutant but there's a few other states that can happen so sometimes the mutation changes the changes the problem in such a way that it's not or this point contract in such a way that's that it's not valid anymore in this case we can compile the contract and we cannot even run the test we and therefore we say beforehand that it is an error mutant and we won't confer it in the later evaluation on the same holds for China which is a little bit differently so what could

happen actually when you introduce a mutant is that you create and loop that runs infinitely long or for a really long time and distinguishing between the tooth really difficult so we say that after certain time out we assume that there's an infinite loop and we won't account for a in the evaluation anymore the last category is in brackets and I did that because it's not really a category on its own it's actually a subcategory of a live mutants and the mutants that be categorized as equivalents are those that don't actually differ in meaning from the

original program so what can happen actually is if you have a mutant you can change a little bit of the code without changing the actual meaning and I've done a little example of such an equivalent mutants here so this is the max function from observing solidity and as you can see we change the comparison operator so that it behaves the friend when's in VR equal felt as you can see when a and B are equal it doesn't matter which one you return for the maximum value because they are equal of course

also in this case we say the mutant is equivalent and you shouldn't say like the test feed children fail on this because it's not about actually it's just syntactically different code so once we have this categorization of other mutants we can come up with a metric that evaluates the entire test feeds and the expression for this is visible on the slide and in natural language this basically tells you what the changes that your test detects a mutant in the code and if we

assume that the mutants that we created represent or a representative of actual bubs then this number will also tell you what the actual chances or estimate what the changes that your test suite will capture or prevent actual bugs in your code of course a hundred percent mutation core doesn't mean you will never help out unfortunately we cannot guarantee such things using our just testing techniques the nice thing is we can have some guarantees and and

evaluate the test be done improve it so first improvement is the next step and at the very base what you can do is changed ensure that the test fit can only improving quality and never decrease by adding like a mutation testing tool to your pipeline but what you can also do is look at the specific mutants that survived and look at the specific areas of the code where these mutants were able to survive and then add tests so that its first commands the main theory and most of the core

concepts behind mutation testing and now I'd like to go into first the mutation testing to adult and how you can use it install it and everything and then the current state of it what the current biggest features are followed by kind of my road map or things I would like to add for usability and performance so yeah let's get started with that I'm assuming most of you would have solidity and truffled installed already as well as Python and assuming that is actually the case even thing you'll have

to do change though vertigo is run this simple c'mon following which you need to say that this network because as you might have noticed in the previous slide we need truffle vertigo is currently built on top of truffle in the future we might extend this but currently it's just truffle in truffle need to have the Ganesh network or it does not work to evaluate the unit I shave Britain once we have this test network so that we also need a char --get for our analysis

and in this case you you admins will just create a directory and and download the contracts for you and we'll also need to adapt the configuration a little bit we need to help the network we just spun up and if your configuration looks a little bit like this you'll be fine if you have your own project which you want to run vertical on and that's awesome and you'll just have to add the network you previously created maybe you have

the network configured that would be awesome and with business over configured we can start the analysis by running this simple command in the directory where you have your smart contract project your toefl project and you start by just doing in vertical run I mean you need to specify the ran network traffic suppose you use for evaluation or you can use multiple networks in which case vertical will do run the test fit an evaluation of the different mutants concurrently but in this case we'll just use one network and note that if you use a different name

for your network you obviously need to use that name instead of development which are used here so if we run this command we will get the following results so as you can see at the top of the slide there we generated five Newton's or vertical genetic mutants purpose smart contract project and we were able to kill four of five of these mutations and ones to fight the bottom of the output you'll be able to see exactly which mutants were alive

and unfortunately you cannot see the entire hole but this mutant was introduced but this is part of the the the value transfer at the meta con contract and as you can see we apply the mutation which changed the comparison between the balance of the sender and the amount that's being sent so this is the sanity check whether you have enough money to actually perform the transaction and actually was be able to determine from this the test for don't actually check the edge case where you have exactly enough money to perform a

transaction and therefore if we introduced a bug at this case people will not be able to detect it so let's improve the test so that we this is the fix I'll start out by saying how the mental coin contract in the truffle box is deployed by default in its constructor it will assign 10,000 a balance of 10,000 to the Creator and in this test the Creator sends a transaction to some other account and initially or in the

base configuration it will send 10 which is not exactly the same as the balance of the creator of course so if we now change this amount to be exactly equal to the dependence of the Creator will we will trigger or we will cover this edge case which was previously in comfort and if we then run first go again on the test suite we'll see that we actually do capture this extra this mutation now and kill it and so now we have a hundred percent mutation score which is really good and by doing this

mutation the cool thing is we have not actually changed the code coverage but we have improved it does feed by covering a dish and additionally educate of both where the sender has enough money to do a transaction but also when it has exactly enough to perform a transaction so I would encourage you to try this out and maybe give me feedback on how I could smooth out certain parts of this process and especially see if there's points in your test feed that might be improved and then I'd like to go over to the state of vertigo itself and the

biggest features that are right now and to do that are one of the system thing about optimizations and features that could be implemented for mutation testing specifically optimizations which can be categorized into two basic aspects the mutation testing can be optimized to do less work or it can be optimized to do the work it has to do more efficiently and this can help put a frame of reference to to the rest of their discussion all right so I've picked four of the features that in then

are in vertical today which I'm most excited about all which I can have the biggest features and the first one is parallel the station or parallel execution of the mutants so as I mentioned before you are able to pass multiple networks to vertical and then it will use those different networks and what it will do is take the different mutants and run your analysis in parallel on the different networks which speeds up execution in a lot um there is basically linearly linear scaling with the amount of processes you use assuming you have enough course of course

and this yeah this this provides major speed ups of course if you have an ice machine this second one is mutant sampling which is actually a fairly straightforward approach so what we'll do there is instead of running the mutation analysis on the entire district we take a sample of the mutations that we generated and only bring the enough on this sample so what we'll get is an estimation of the mutation score that we use the entire set of mutants and which has a little bit of variability so it's

not super accurate Petronius channelized part of the mutants and thereby speaking of the analysis a lot is 12 so we can do like a sample of 10% speeding up the analysis ten times and there's the balance there between the accuracy you borned and the speed is required [Music] Oh George no feature this is actually a feature is randomness Universal mutator here so Universal mutate there is a another mutation to some project and what they did is on the final generic way of

writing mutation rules so that you can easily apply to different languages but it also allows you to do is to do really is the development of different mutation operators and maybe write custom mutation operators so I'll give an example and from the universal mutate or veto itself of what a mutation will look like or few rules actually and these actually moved for first levity is that you can write mutation rules that

specifically introduce bugs that are possibly you need to your project so what you can do is maybe swap out certain function calls which you think he could be done by accident and write this as a mutation row and then by running the mutation testing using this mutation row you actually evaluate how good your test read is detecting that type of that and this is useful both for a developer who's trying to improve recipe but also for an auditor which one's seen know whether to one singer whether it developers actually checked for certain cases without necessarily

reading all the tests and manually exploring all possible mutation all right so that's the benefit of universal mutator provide a the last major feature is compiler equivalents and this kind of reflects back to the iteration which I mean we shouldn't not towards the beginning of this talk so to recap a little bit an equivalent mutant is a mutant that doesn't actually change anything about the code and we can use the compiler you'll see to

capture some of these so what I'll do here is take the original program and its mutant and compile them both using so see so the mutant is applied on the source code level not on a byte level and once the two contracts are involved we will compare the byte codes I'm assuming that so she doesn't have any part we can know that if two two byte codes are the same that there is no difference in the meaning between the two contract they are semantically equivalent and what that means is that

we know that there's no actual difference between the two and that the mutant must therefore be equivalent or any present mutant these speeds up the analysis because we won't have to execute or catalyze the equivalent mutants anymore but it also speeds up you emanuelly where you need to go through the alive mutants and then see if they are equivalent or not as so this makes it a little bit more useful in that perspective

all right so what's next on those two aspects that I really wanna tackle in vertical the first one has optimizations so speeding up the analysis to give you like an even estimation on the current performance i've run experiments on projects like argon OS and alvin supreme solidity were issued between the analysis to thirteen two and four hours but i really wanna be able to speak their lap into like ten minutes or 30 minutes which is a lot more useful as well the second aspect is usability

decreasing you like setup charm and making it easier to used mutation testing using different frameworks other than truffle which is currently supported so different optimizations let's all buy into incremental valuation so here the basic assumption is that if you run our mutation analysis in your pipeline or frequently on your code base between those analyses the killing the test that kills certain mutants will stay the same a lot of things will stay

the same so you might be able to reuse this information so what we can do for example is for one test record which mutants are killed by which tests and then for consecutive runs first ring these like killing tests on the mutants if they are still present and then continuing with the other tests mr. sweet so here like the optimal improvement will be that you only have to run one test per mutants which would create a major speed up the second optimization is Nugent

clustering and what weeds do here is actually pretty similar to the mutant sampling so in mutant sampling what we did is we took a random sample of the mutants with mutant clustering we will take a more sophisticated approach where we create different clusters of mutants that are similar and then from each cluster we'll select one mutant and this is this will allow you to achieve the same benefit of only having children or evaluate less mutants but it will be more accurate than just taking a random

sample of all the mutants and the last optimization is called test selection and this is actually the most straightforward well instead of running the entire test suite for each mutant we will take our code average for each test and then given a mutant we'll only if run those tests that actually cover commute so assuming that you don't have tested covers the entire contract that not all your tests cover the entire project at once

this can also provide major speed-up the only issue here that as far as I know of the current code coverage select equal coverage implementations do yet support reporting this fine-grained coverage function that I believe this is something that's on the roadmap of some of the projects which is bang the second aspect which once or needs some improvement is a dynamic or disability or

and the first aspect there is again the Ganesha network creation and so as you saw in the beginning of of this section on on vertical you have to set up a truffle network or a Ganesha network and we had to configure it in the proper configuration and what I'd like to have is a situation where I'd wear vertical will dynamically spin up Ganesh networks and Institute configuration so that there is basically zero setup necessary to run mutation testing the second benefit that this will have is that did resolve some issues we have with Ganesh

which is basic which is caused by the fact that finish probably isn't made to run a test feed 500 or a thousand times which isn't all that weird but during our experiments we would often create hundreds of gigabytes of data and millions of files which put a strain on our file system and hardest which of course we would like to prevent for for common use cases so dynamic a national corporation will allow us to really spin up thing memory databases about the pen databases on your hard drive which will

be a big improvement second juice ability aspect is framework expansion so I mentioned this one before right now Rio needs to pour truffle but there's a really nice projects out there that are getting more adoption like brownie and embark which I would also like to support so that more developers can can actually use the tool and virtual would become more accessible so in conclusion we talked a little bit about mutation testing theory the generation evaluation of an evaluation

of mutants and how you can use it to improve shit-ass repeat then I talked a little bit about vertigo itself so the tool implementation how you be able to install all and usage the present state of vertical what kind of features we have right now and the like the lowest hanging fruit and the world map prefer to go in the future and with that I'd like to end this presentation and thank you all for listening I think there will be some time for questions but we'll have to figure that out a little bit hard to do that using zoom

yeah thank you we'll try to do some question do you hear me yeah so does anyone have any question if you do with I'm just going to ask you two to come here to ask it to have it recorded and for Gen 2 to be able to hear you could you return it Simba's no question okay so thank you very much we're going to have the next tool quiz Bernard who

was consoled before said we just changing the schedule so if you want to stay after this one you can there's something it's not expected but there's something for you Thank You Joran you