100% Test Coverage but 0% Security?

Transcript

hi yeah thank you very much my name is Felix uh from open Zeppelin you might know our security library of smart contracts and we of course also do uh audits monitoring and many more things and today I want to talk to you about testing so um we have like one really crucial question to answer if we get 100 test coverage do we get some security no security a lot of security I'm trying to answer this today so let's take a step back why are we testing at all typically you want to see um whether your code base actually has some kind of functionality probably the

functionality you intended your code base to have that's kind of the point of tests but then looking a bit further you have like additional points of your tests typically you don't only want to do unit tests you want to do like some integration tests or some end-to-end testing and they actually have a descriptive functionality so you can really see like what's the happy path through your code base what are like the interesting ways a potential end user can interact with your code base so this kind of descriptive functionality is also in there and then does it contribute to security well we are hoping that it does but um we don't know at this slide so let's let's move on and answer this question

later now talking about the quality of tests after a while you want to be done with testing so we have like different quality metrics um the first one is a functional metric it's the test coverage of your code base and this is typically like a line coverage or Branch coverage it's a nice number and we're assuming that it's like 100 in our test example so we're doing very good on on this case and then um again descriptive means that you have kind of interesting test cases not just like very small unit tests but some very interesting end-to-end test cases and now it's again the question do we need an

additional metric or are we just like accidentally already done with security and have perfect security as well so let's look at a small example this is like a very small elc20 token I wrote you can see above in the slide 100 test coverage is ensured To Us by like Foundry and you can see in the lower part of the slide well I wouldn't really call it interesting but because it's a minimal example it's kind of like the three use cases I imagined for this token that you can mint it burn it transfer it yeah they all work out they're all covered in this test case so basically we're doing perfect on the functional and descriptive side so are

we secure no this has a public burn vulnerability so you can just burn any user's token and basically this is like a critical vulnerability and the security here is really like zero percent you need to fix this you can't deploy it in this shape but so what is missing this test case is missing um if you don't know The Foundry syntax you see in the name that it starts with test fail so it's kind of asking for the opposite of what you see in the test case it kind of verifies that this test will fail and um yeah it's a typical test that just uh tests that like an arbitrary address like a non-owner

address can like burn a thousand tokens of another user and yeah you really want to make sure that this cannot happen so you want to write this kind of negative test case now this was a small example let's move to the overall principle what we typically see when somebody says they are doing testing is completely on the left side they do functional testing they are concerned with coverage but this is just one aspect of testing and it's good that you do it but it has actually very little to do with security if you want to do security testing you have to be on the right side and you have to ask yourself like how much testing of of an absence of functionality am I doing so for example

for unauthorized functions am I really testing that like a message sender that is just an arbitrary person cannot interact with a contract in a certain way and so this testing for absence is what we call security testing here but now I want to take an additional step back and wonder about like how does testing look uh in theory and in practice so in theory um numbers provided by IBM is that yeah we should invest a lot in testing because it really pays off if you find something early during the design phase the implementation the dedicated testing phase your cost is kind of moderate and it really deploy it really explodes in

the deployment phase and this is not for blockchain systems where like the number 100 you see might be even higher because of like the tvl attached to it but this is always already like across arbitrary software systems So in theory everybody should basically do the best possible testing because the economics clearly tell you it will pay off now let's let's look at the practice testing is not sexy testing is really like uncool nobody says oh I'm really passionate about like QA testing I'm a tester um like all the cool stuff is basically associated with hacking so um of course

for like your protocol that's kind of like a bad thing right like you don't want to be hacked but hacking is rather cool while testing is um kind of like a the ugly thing you have to do but no one really wants to do so let's now look at how we can like fix that so this is my recipe to make testing attractive basically do hacking as part of testing so to do like really good security testing we start with while true because you can iterate this Loop forever what you do is you come up with like a creative POC exploit you're basically trying to

exploit the your own protocol you're trying to develop then you verify that it doesn't work hopefully or you actually found something that you really need to fix and then you just don't throw this away but you integrate this as a test case into your security testing test Suite and that from like a day-to-day activity of how you think about it makes this a lot more attractive because this is not like the unit testing or the boring kind of testing you you know and it's it's like the typical University curriculum but this is more like the modern kind of like hacking mindset you apply you just apply it to like an artifact that makes it into your test Suite

so nice Theory but now to actually add more practical points to this uh let's look at the projects just this year that had like some code level vulnerability so this is no exposure of private Keys No Rock poles or something like this rather they had a bug in the code that could have in principle be prevented by this kind of testing methodology and we see that like the funds stolen today in this year are 250 million already so I think this is like a good argument to potentially integrate this kind of like saw the security testing into the test Suite so now on a very high level if you are a

decision maker you don't want to like recall the details here you want like a really simple metric count the number of interesting test cases based on proof of concept exploits in your test suite and make this the metric of security testing so to summarize 100 test coverage is functional testing it's not related to security if you want to do security testing which you should you should test the absence of something if you want to do this in a good way you need to like adopt a hacking mindset and really do exploit pocs as part of your test Suite and unfortunately there's no Silver Bullet this is not exhaustive you can

start this as uh as part of your project internally and then best reach out to your security partner to basically further enhance this kind of testing methodology thank you very much [Applause]