Mutation Testing with Slither - A New Way to Find High Severity Issues

Transcript

Okay. >> Well, hello everyone. Thanks for coming. I'm Guillermo from Trail of Bits. And today we will talk we were we are going to talk about mutation testing a way to find bugs that the tests are missing. So, we already know that tests improve your code and to quote Alex, he said that if your code is not tested, it is broken. So, we need a way to figure out how to test our tests. So, normally for a blockchain project or any project in general, you will have different kind of tests. Unit testing, for example, that test the behavior of individual functions.

You can create unit tests for edge cases, for example. Then you have the next step is integration testing where you start testing the how the different contracts or different functions interact between themselves. And how the whole system works. You can create user flows, administrative administrative flows, or whatever. We also recommend that projects that have some kind of arithmetic or rounding error that are prone to rounding errors also implement fuzz testing to ensure that you detect those kind of issues.

But the big question here is when you have your test suite implemented, how do you measure how effective the test suite is to catch bugs in the real life? So one answer to this would probably be coverage. Is it a useful metric to know how how much your tests are covering your your code base? But the thing is that coverage doesn't really explain or doesn't really show you what uh bugs are being missed in in your in your code base.

So, for that we will be we will be talking about mutation testing. First, a brief description of what mutation mutation testing is. So, basically, in a normal project, you have your code base, you have your test suite, and when you run all the tests pass, and that's the minimum expected for a project to be safe. Uh in the case of mutation testing, we are going to introduce mutations to the code base, and those mutations can be small bugs that are introduced purposefully, like uh on on purpose, to change the code base and see if your test suite actually detects the change and make the test fail.

Those mutations could be comment out a line uh replace operators, like assignment operators, uh arithmetic operators, or whatever. Uh replace conditionals inside uh the if conditional or for loops, for example. There are a series of many different mutators that you can uh introduce. So, for each change that you make to the code base, you basically generate a new mutant, and you run the test suite against that mutated code base. Of course, we can have two possible outcomes. If the tests fail, then your test suite is actually detecting that bug that bug, and everything is okay.

The that modification, that mutant was caught by the test suite, and you are safe from that particular mutation. The alternative is that the test pass, and now you have a problem because you modified your code base, and your test still pass. So, that's when you have to start uh investigating and see why the test passed. Uh this is a very time-consuming process, as you know. Uh mutations are introduced one by one, uh each file and each um, line of each file will have to be mutated. So, the idea is to automate this and not do it manually, of course.

So, for that we have Slither. Slither has a tool that's called Slither mutate. And it's supposed it's meant to be easy to use and it's supposed to be used by every project that has tests. Uh, to run Slither mutate, you point it to the folder where your contracts are and you specify which test test command are you going to use for running. For example, we have forge test if you are using foundry, hardhat test if you are using hardhat, or whatever custom script you you you have to run your tests. So, what this will do will start generating mutants, running the test against the mutants, and detect if some of those tests

uh, don't pass with the mutant. Um, basically, the output for this tool will be similar to this. For each contract, each file, each mutant that it generates, it will show that it's mutating the contract name. It will show which mutator it is using. Uh, and it will show the previous uh, line the the previous code and the new code. Uh, and it will show you if it caught the mutant or not. So for uh, big code base, this will take long time to run. So, what we suggest for auditors is that you start at the beginning of the audit to run the mutation test. For uh, other for normal developers to run this once in a while to ensure that the

test actually cover everything. Uh, to finish, we have the help command and the list mutator command that you can show. And this is a blog post that we made about the tool with some real life examples. Thank you very much.