Enhancing Test Coverage: A Framework for Effective Protocol Security

Transcript

so hi everyone my name is Dimitri and I'm C at mix bites and today I'm going to present my experimental framework regarding how uh you should write test to catch some severe bux uh before I start let me share a bit about mix bites uh so we're mostly focusing on conducting security Audits and research uh we have been working since 2017 and since that time we worked with more than 60 clients and performed more than 250 audits most of our clients are coming back to us with uh new partical versions

and small updates in their code base uh so the list of our key customers are presented here so now back to the topic uh the general recommendation for writing test is often unclear for example preparing 100% test coverage is not save you from severe bugs also a general recommendation for using a hacker mindset while you writing the test is uh very hard to adopt if you're not a whad hacker or experience security researcher so today we'll discuss the question presented here and the first

one is will 100% code coverage prevent hex and uh answer is of course not the second question is what exactly is a hacker mindset uh and the third one is is there any other type of mindset that will help writing tests and the last one how should writing test be approached to identify as many issues as possible before an audit uh before we go to next slide how many of you you heard about a hacker mindset please raise your hand and how many of you know how apply to this recommendation okay it's nice uh so the

framework like it's our experimental work and I went through a lot of different security audit reports and bugs that happened in our industry and uh came up with an idea that actually while you're writing the tests you should use three mindsets to cover all possible severe bugs the first mind mindset is a hacker mindset and we will discuss it in detail later the second mindset is a formal verification mindset or it also can be named as a inent mindset and the last mindset is a system architect mindset uh so applying all these three mindsets should help you write very

thoughtful tests uh in our company we promote an approach for writing tests which as follows uh not writing test for audit uh the audit is a process of generating questions regarding the code and answering them this approach is much easier to explain to new security researchers and web 3 developers with this approach it's much easier for you to control the progress of your audit uh and extremely help helpful with the code bases which are well written and like huge code bases around 8,000 lines of

code so I think that's it uh that's why it's easier to explain this framework as a set of questions for each mindset uh and uh like the answers for this question should help you write uh good tests that will help you identify bugs so to use this framework you should select one of the mindset one after another uh get the questions from this mindset ask them yourself and uh like the answers will help you generate the test scenarios that will help identify bugs so the first mindset that I want to discuss is a well-known hacker mindset

and the like general idea of this mindset is to force yourself to think about your code not as a developer who developed all the use cases for users but as a malicious user who is trying to uh find a way to interact with your protocol to break it so instead of asking how is supposed to work you should ask yourself how it's not supposed to work and that's it uh well no uh the questions for this mindset are follows what uh use cases exist for the protocol that should not be accessible to regular users and this question should be uh asked for all the

public and external functions that you have in the protocol to identify which of them should be called only by some uh special roles or special services and uh like write test that check that this function can be called only by a verified Ro uh for example here we can come up with a with a test that check that the function can be called only by a offchain service that should update some parameters the second question is what limitations are there on the sequence and frequency of function calls by users and this question should help you

identify all the reentrancy attacks and uh it should help you to restrict the user flows which with which users can interact with your protocol so you have to restrict all possible use cases that you designed before and all other questions should be like restricted for all users and it should like uh cause revers so here an example of a test for example you have a function that should be called only once per an hour and if it's called like more frequently then you have to restrict This and like it can be done by Ed air chire and uh

proper test should be written for that the third question is what constraints should be placed on user input parameters uh and this question is about restricting your protocol even more so the general idea actually of of a hacker mindset is to restrict your protocol as much as possible and this third question is about restricting input parameters and here we could think about a function that accepts unsigned in 256 and converts it inside the function to unsign in 1 to8 and if you not restrict the maximum value of input parameter then your protocol will be probably hacked and the last question is which

parameters cannot be controlled by users when calling the protocol and this question is about identifying which parameters shouldn't be controlled by users and should be controlled by some white list or by some constants that only admin or privileged role can update so uh the next mindset is the formal verification mindset and this mindset cannot be merged with the hacker mindset because here you should ask yourself about not how your protocol shouldn't work but you should think about how your protal should work according to the specification and for these purposes you have to write

specification it's like a very hard task nobody likes to do it but you have to and uh the spe the specification will help you a lot with this mindset because you will know like what to expect what to expect from your protocol and you need to write test that checks that it works according to the specification uh and so yeah it's like identifying invariance from the specification and implementing special tests like fing tests or going through formal verification to check that your protocol Works according to it and here are some examples of the invariance the first one in a function

that calculates the number of tokens to be output during a swap it must be guaranteed that the number of tokens output cannot exceed the contract balance this invariant is uh like very helpful for new version of unisoft V before when like only one contract contains all the tokens but there are a lot of contracts that could interact with this contract and if they somehow trying to transfer more than uh they can then most probably the protocol will not uh work as expected and the second invariant is the variable that stores

the total amount of tokens that the protocol control is less than or equal to the balance of that token for the contract address this EnV variant is uh really good for landing protocols or just regular walls that accept some deposits from users and stores them in special variable and you should check that this variable is uh like the value of this variable is equal to the actual token balance for this contract the last mindset that I want to discuss is a system architect mindset and this mindset is about thinking about all the protocol integration in a very

paranoid way so like all the external integration could fail at some point uh like an oracle service can stop updating the price because it reached some threshold value and this happened with Luna I think when the price goes like really uh fast uh a offchain service could uh have a bug in a new update which uh leads it to misinterpret milliseconds as seconds which can affect your protocol external def5 protocol can update its logic and uh distributes uh more reward

uh to your protocol and this reward should be claimed to a special function that you didn't add because you didn't know that you should uh and the cross Chain Bridge uh can just fail to transfer your uh message to another chain because you failed to set correct G amount or there was an internal error in the bridge so the here it's only one main question that you should ask yourself it's which of the protocal independencies are external and could fail and it's very easy to answer this question you should just go uh through

all the Integrations that you have and mark them and after that you should think about like in which way it could fail your protocol like what can happen with this integration and uh I think it's crucial to prepare mocks for all of Chain Services that will communicate with the contract to check cases of how the protocol will react on a failure in the offchain service and another question here is which of the integrated Protocols are upgradable and here you should just think about upgradability feature of the integrated protocols and if you integrate with such protocol it's better to make your contract upgradeable also so in conclusion I want to

emphasize that test preparation is a very important thing and like developing a web 3 protocol without uh thoughtful test preparation is dangerous 100% test coverage uh is uh not like very good uh recommendation but it's should be done you should have at least from 80 to 90% test coverage and you should cover all unit uh and integration tests so you should be ensure that your protocol works as you expected and after that you should apply this Frameworks that we discussed today to ensure that all severe bugs are covered with the test so while you're

using this framework it's also can be beneficial for you to prepare a checklist of the test scenarios uh so when you will be developing developing a new version of your product protocol you could use this checklist to prepare tests that you prepared for the first version and also all mindsets are crucial and it's important to cover your code with all the mindsets and uh like you can skip the system architect mindset if you do not have any external integration but uh I cannot like remember any protocol that I audited in a year that didn't have any

external Integrations and uh yeah write test stay safe uh thanks for your attention uh I would be happy to answer your [Applause] [Music] questions yeah we have some time I think hello okay uh thanks for the talk uh usually when trying to find um interaction like um sequences of interaction that lead the system in the broken State um like writing a state machine diagram works

pretty well do you have any tips or any automated tools to do that kind of stuff could you repeat the first part of the question it was um so it's the idea of like drawing a state diagram um to try to lead the system in a broken state do you have any tips or automated tools to do that kind of stuff I don't know regarding the automation tools here like most of them are not very good right now uh but yeah like right in a state machine for your protocol it's very nice thing to determine like in which St it could be and just like restrict the states that you want your protocol to be

so yeah it's it's a very nice thing to do more questions uh I have a question about the different kind of uh uh test which part should be covered of the business logic my point is um you have mentioned about unit test integrated test even those kind of formal verification test uh I I say that

a kind of test so according to uh uh our own business for example Landing protocol which part of the specific uh test should uh cover which kind of you know different uh test method or kind of patterns do you have ideas ideas or recommendations to for uh the the protocol Builders how to decide and how to uh consider that that is um we can satisfy all the the Logics we can

cover you mean like uh for which functions each mindset should be applied or yeah uh you you are talking about the coverage mhm the the you I I consider that's what that was a a unit test coverage all the code of the the uh Branch uh just like basic tasting to ensure that your protocol just works yes how about formal verification what uh what kind of integration uh test or formal verification should cover about the business or the the basic code base so unit test and integration test like like General test coverage it should help you to be sure that your

protocol just works and all the mindset that I discussed it helps you to ensure that you have additional layer of security on top of just like basic use cases so you have to write basic tests and after that you have to switch to the mindsets and cover like with a hacker mindset cover with a formal verification mindset and with system architect mindset and how about uh fuzy testing fuzzing like actually it's about the same with the formal verification but there is a small difference that formal verification gives you 100% sure that the invariant holds for all the states of the protocol and fing tests only some

specific cases so formal verification is better but if you don't have like time or budget for this you can use fing test okay thank you okay