State of the Art of Ethereum Smart Contract Fuzzing in 2022

Transcript

hi everyone so today today i'm going to talk about the the actual state of the art of eternal smart contract fuzzing so my goal with this talk is basically to give you an idea oh exactly uh neteron small contract further work what's the key features that make offers are good further and uh ideally to tell you a bit more about the actual tools that you have on the market and which one are for me personal opinion the best and what can be done to improve them so a quick um intro from patrick i'm the founder and ceo of fuzzing labs so as the name suggests we are doing fuzzing full time

so fuzzing and vulnerability research mainly on layer one and two we are also providing training regarding rust go security and so on and maybe you learned about one of my main projects that was begun first the atom 2.0 differential further for consensus clients so what is the actual state of the art so first of all i need to explain what is fuzzing just to be sure everyone is aligned so fuzzing stand for first testing so it's actually the most efficient technique to find bugs on program in general the main idea is to provide a random or pseudorandom input into a program monitor if it's

crashing and if it's the case it's it's a good point for you there is different type of further the first one a different approach um the first one is black box so you don't have any information about the target the second one is gray box meaning you have a bit of information of maybe what kind of input you are taking inside the program and the last one is white box fuzzing meaning you have the source code of the program and you can basically modify the program recompile it and do some other stuff so the way it works usually is you are providing input you are executing the target you need to find out if there is any crashes or you can have like detectors that will detect

something weird is happening and then you will do that over and over again like a big infinite loop that is doing the stuff so over the time we saw multiple servers so i i made a quick timeline uh one of the first one was echina then we got some um further like contract further chances like almost every big security um ethereum based company um i've been creating his own further we also have rv uh so i put it in 2020.

to be honest it was more in 2018 it's just that they release some like public services at the time and you will see later it's close source recently we got foundry that support fuzzing i will discuss more about that and a quick shout out to a fuzzing like a d-gen it's definitely not a really efficient further and so on but it's really nice that people start to just try to publish some new stuff and make like learn more about fuzzing for people in the ecosystem so quick overview first of all which one is open source so almost all of them except harvey and you will see that it's a recurrent stuff then we have the language of development

that's something i think is really important because if you are writing the stuff in a language not like ascal for example you will also have people that will learn more about the internal of the code and maybe try to improve the stuff and it will basically make it better and better over the time so a bit red flag for h9 in that case because i don't know how many people are familiar with ascal but at least not me then we have rv i don't know unknown we have some go we have some rust and also some python so for me that's the the main language that we should develop further in in terms of documentation um

we have a china and foundries that are really well documented and it's even even more in the future i think and the other one not and the last point is is it maintained or not that's something really important especially if you want to start to learn how to use a further and this other will be updated over the time fitting with like the new upcodes and the new stuff that will come in the in the ecosystem and in the smart contracts so that's also something that is really like a key point uh that's where you see that contract further enchanters are actually not maintained anymore so deprecated but there is still some interesting stuff to learn from them so let's see a bit more about the

further internals or exactly a further work so the first feature that you need to have is the input generation in our cases the inputs that we're going to generate are basically transaction and there is plenty of stuff we can generate from that so the first things to generate of course is the transaction itself and what we call the call data so you can perfectly leverage on either the solidity source code the api to basically learn more about the type of argument and generate the transaction that way but even if you only have the byte code that's it's the case if you are doing like black box fuzzing you can still leverage on that and learn from directly from the bad cut typically you take a look at the

four bytes um the function id the method id um directly inside the bytecode and you will be able first of all to find the method id and also you can do like use some reverse db database to actually find out the parameters so as you can see in this example right there we if we want to trigger the asset we will need to generate the proper value for a and the proper value for b so that's typically something that the further will be really efficient to do and we will be able to generate pretty easily then we can generate some sequence of transaction that's something really

important and not done by all the others the main idea is since after each transaction you will modify the internal state of the of the contract by doing a sequence of transactions you will be able to trigger some part of the code and some state that was not discovered previously tiny example is the famous rubix contract and in that case if you want to exploit this contract you first it was needed for you to call the dynamic pyramid uh function and then collect all fees so if you you were just calling one function at the time and just reset everything and reset the state of the smart contract that's typically a vulnerability that

you will not be able to find automatically by using fuzzing so regarding our competitors right there we can see that most of most of them the requirement is actually to have the source code in order actually to determine the method id and so on so that's the simplest way only one of them was using only the white code it was a contract fuzzer and what is a bit more interesting is the input generation what you can see is almost all of them was actually generating both and using both a technique for input generation so both the cool data generation and at the same

time the sequence of transaction and uh what you can directly see in that case is that foundry for the moment is only generating a single transaction so that's where you see that there is we start to have some limitation in the different furthers available and that's definitely a point that should be improved in the future another really important feature is actually the execution engine so we are generating transaction we are giving that to the evm and the main question is does does this evm is part of a local network that you are running or

is it possible for you to directly provide that to the evm itself and get more execution speed that is one of the key metric when we are discussing about fuzzy so in case of local testnet net you can easily implement that using ganache for example and you will basically have your internal network and you will provide the transaction usually using the apis and the libraries that are provided in ganache i don't remember the name but whatever and uh that's okay it's working really well it can be done really easily that's typically what fuzzing like a djinn was doing it was further developed in like a four hour

during a twitch stream and basically that was the technique he was using what is way more efficient and way more interesting is directly to provide the transaction directly inside the evm so you will only execute the evm alone and the main purpose for us will be to get way more execution speed because you will provide the argument and it will be executed you will keep the state and and go on and go and so on so you provide directly the transaction to the evm you execute the contract you get new states and it's way more faster so what we can see there is that echina

is actually using hevm so written in scale as well so that's maybe a reason why it should not have been developed in scale in the first place and then uh we have some um um some puzzles that are actually using like a local test net like chamfers rv and fuzzing like a dj for foundry they uh implement their own rust version of the evm and that's also why foundry is really fast and really interesting for us because we are getting way more speed using that the third point that are really interesting is the detection mechanism

or do you find or do you identify a bug so um in that case you will need to do that directly at the evm level you need to have a way for you for the virtual machine to tell you okay there is something wrong that happen and you need to actually notify your you as a user that's something wrong or occur so you can perfectly first of all leverage on all the exceptions that are directly inside the vm so that's the exceptions that are actually detect at runtime by default by the evm you can think of when you are using stop reverse self-destruct all those kind of upgrades you also have all the stuff like out of gas illegal instruction

assertion failure and arithmetic overflows do stuff are detected by the evm so for us is directly detection mechanism for free because it's part of the evm design another point that are asking as a user um the most common one is actually to do property testing so what most of fuzzing framework on a term actually asks you to do is to write what we call invariant or at least testing function so that means that means you will need to write in solidity some functions that will be there to test some particular

properties so that's really nice but that requires some manual job for you to to do you will need to actually understand the smart contract write your own property testing and based on that you will do the fuzzing and expect maybe an assertion to fail if something was happen so of course almost all the further actually leverage on the runtime exception because as i mentioned that's some free stuff for for us and for the property testing it's a bit more particular it's really depending on the design of the of the further

as i mentioned if the fuzzer is actually taking some source code as like leveraging on the source code you compile the source code further further and then the fuzzer is running on top of that that's also why they need the source code because they will take in consideration the function the testing function where if you are only using the bytecode you will not get any testing function directly like on the magnet so that's mainly why so you can write those property testing in different language depending of the further mainly solidity and it totally makes sense and rv is also using something called scribble where basically you define

on just on top of the function which stuff shouldn't happen and they will i suppose convert that into something else as i mentioned um rv is closed source i actually request to get access but it was too late uh to to test in before the training before the the talk so um i'm only and only took a look at the documentation and all the paper related to arvid to actually find out what they i suppose are doing then there is some other features that are interesting you can think of code coverage corpus replay and gas usage so i will try to to detail a bit more this

one they are not requisites to build a further but that's some extra features that actually bring a lot of value when you are doing that the first one called coverage so your goal will be to monitor which op code basically are executed during fuzzing so that's a really interesting metric because you can directly flag which kind of function you have not reached at all um and as a analyte and as an analyzer to the result you can determine okay i need to improve my fuzzing and maybe i will need to add some extra testes to actually fill the gap and go deeper in the in the coverage so uh etching and harvey are doing that

for the other one for what i i saw foundry is doing that more or less partially or it's a current development feature so i i suppose it will be integrated later um in the same way you can have code coverage at different level either it will be at the source code level or at the bytecode level usually since they are leveraging on the source code they are able to provide you this information at the source code level but it's actually it will actually make more sense to do that at the bytecode level to be more accurate of okay this specific op code have been executed and this one not because i mean not everyone know exactly what the solidity compiler is doing to their code except if you are doing analysis of the evm bytecode

for the corpus and transaction replay the main goal is you are running your further so it will generate a lot of inputs so it totally makes sense for you to keep them just doing that as a one-shot stuff it's really important to store everything that have been generated and in the future being able to basically replay everything and not losing all those execution computations that you have done previously something else that you can actually do is if your smart contract is actually public or if you push that on the on the testnet you can perfectly extract the previous transaction to the smart

contract and put that into your own database of of in your own corpora that will be something really interesting uh for you as well because you hope the further to actually to do scopora do some mutation and manipulation of that of that and replay the stuff so that's some extra stuff for for us to basically be faster and go deeper into the into the program so again i will let you check which one are doing what uh for foundry uh they're not doing that for the moment but i think it it's a matter of time at least it's not that complicated to implement finally uh we have a gas usage report so

you can think it's not something we exactly want to take a look if you are doing fuzzing or maybe only taking a look at vulnerability but actually in some cases you will have vulnerabilities that will maybe loop to itself and just exhaust all the gas that you have and uh it will not be something that will be it will be detected only if you are triggering maybe like a runtime error like a gas exertion but if you are just before the gas extraction it will not be something you will notice so in that case having a gas report and be able to know that okay this specific sequence of transaction is actually burning a lot of gas can also indicate

you that you need to improve um the the code on that part or maybe you you will find a way to basically reject the transaction or actually prevent this stuff to happen before so again some further are actually doing that so main question which one to choose so that's the global summary of the stuff um of course you can see that some of them are really green and that's that's nice especially etching rv and foundry that are for me the key constant in in this way in terms of speed we can see that hina

and fondry are actually really fast for rv okay i don't know and that's why i decided to put on the first place uh hmr for me that's the uh that's the best one uh that's the uh it was one of the first ones there is a lot of documentation available uh it's really complete uh as you saw is actually feeling a lot of the the requirement and it's well maintained and that's something really important what i think is the most promising one is actually foundry it's really fast there is a huge community an active community on it um and um what i think should be a really good feature in uh in the future will be

to add a dedicated first sub command for foundry for the moment you are using foundry like doing foundry test and it will do fuzzing for you and as i mentioned only a single transaction fuzzing on the on the different function of the smart contract typically i think the best new feature to implement will be the generation of sequence and also having um the the first subcommand to allow people to only use fuzzing and to exact put that on the ci doing first testing like it's done currently you will maybe do 20 execution 20k execution on each function

but it will not be enough for you to actually find some in deep bugs the last one based on all the paper i read about the the subject seems to be rv as i mentioned close source so i was not able to to verify uh that it's actually really working well so i will uh expect it to to be the case um but yeah if you are looking to add even more fuzzing and just be more secure it definitely makes sense for you to give it a try so final question do we need another further i think something that is currently missing is maybe a black box further like you are only leveraging

on the bad code um and you are doing fuzzing only on that the only difficulty you will get is regarding the detection of vulnerability because you will need to detect vulnerability at the bytecode level and you will not be able to implement property testing so that will be the downside but it's still something that is missing and it could be interesting to actually have that in the future all for me you have the slides available directly there and i'm i should be able to take some questions thank you [Applause] questions

i'll ask a question then so fuzzing is obviously really important testing technique do you think there's also a need for some hard-coded cases alongside that to give users some ground truth or what kind of approach have you seen with people doing more hard coded tests alongside fuzz test yeah definitely i i think so it's a good question typically what i mentioned with uh do being able to detect at the bytecode level that's something that should actually be done i think directly by the the evm typically there is a bunch of stuff that we can just create some general

detection rules that can be added to the to the evm maybe in in kind of like a debug mode or something and uh that means it will bring us for free the detection mechanism uh typically like rio francis or this kind of stuff could be detected at the bytecode level so it could be really interesting to actually have that provided by default for people maybe not directly on the main net because it will maybe break a lot of contracts but at least in inside a kind of debug mode it will make sense hi

very interesting talk thank you a lot of protocols use a collection of contract rather than just one contract um how would you recommend fuzzing that like all together as a whole or individually contract by contract good question um so usually in this case that's also why some further using the local test net to actually simulate that and i think some of them are actually even doing more that is uh like cloning the exact state of the mainnet to be sure it will be replicable uh and uh perfectly uh good on that if we want to do that fully locally that just mean we will need to i mean

the evm will do the job in the same way just that you have multiple contracts so um in term of engineering that's not that difficult and it will just mean the execution you will not get like direct overview of the execution and you can actually make it even more simpler with like the generation of transaction because you can perfectly say okay i have like five contracts sorry i have five contracts but i will not generate only transaction for one contract but i want to generate sequence of transaction of all the different contracts so of course it's to be more difficult for the further to do the job but that's some

like real cases and like potential experts that can happen in the wild thank you thank you [Applause] you