Safe Upgrades, The Most Dangerous Game

Transcript

I'm going to take just a minute to set up here but it will pay off and that my speech will be actually be shorter uh and that's kind of what uh one of my topics is putting a little bit effort in ahead of time and saving yourself a lot of time afterwards I'm going to talk about the single most under recognized danger in crypto upgrading a running protocol uh the stakes are massive and mistakes can be mistakes are really easy and the consequences are just bad uh so Ave deployed a broken upgrade and locked up 120 million dollars for more than a week and Ave is the second largest D5 protocol out there last time I checked

it had an 8 billion tlv compounds did one billion dollars a few months earlier and three separate audits and it was frozen for a week and some now I'm not picking on these two projects particularly they both care about security they both did a good job communicating afterwards but safely upgrading a protocol is hard uh and repeatedly we see even good projects do a bad job of this now Lido uh just did a great job of it they do everything right and so to some extent this is just focusing in on uh what you

can do just three things you can do to make your life so much better and not be stupid uh so my name is Daniel Von fange uh you might know me from Twitter um I'm the smart Trend track security guy at origin protocol uh this is going to sound like Shilling here for a second so in two sentences I'll be done and there's an actual point to it but we just launched a new product Oh a few weeks ago and currently we're the number two uh yield aggregator on DFI llama for ethereum our previous D5 product which is a couple years old reached a 296 million tlb since I've been doing smart contracts uh upgrades and gatekeeping or

security and upgrade gatekeeping we've done 50 plus upgrades to these two protocols with zero hacks losses of funds funds freezes or dramas so I spent a lot of time over the last two years thinking about upgrades and I really care about this and it's so painful to see people repeatedly Doing It Wrong which is why I'm standing here today so what's different about new code versus upgrades when you write a new protocol there's just your code it floats alone suspended in space you can read your code and know everything about it and what it does Auditors can look at it and know everything about it but when you're upgrading your code there's your code and then there's all the stuff that's already there

the state data the config data for your protocol there's all the other internal contracts that already exist and then there's all the external contracts all around it's this cloud of existing state and contracts that broke both the Ave and the compound upgrades in both cases the new code that was deployed was audited it was correct in one sense but it did not fit the environment that it was put into this is the biggest difference between fresh projects and upgrades correct code is enough for new code but for a safe upgrade the new code also has to work with everything in the existing world and one more thing when you write a new

system from scratch you're often thinking about the entire system as a whole but when you write an upgrade you're thinking about list you're thinking about as little of the current code as you can possibly get away with to write the upgrade but the reality is when you write an upgrade you actually need to think about more not less because of all the other things that are out there so I'm going to go over three important things you need to do to do good upgrades uh the first one is simple and that most people here probably know about this one raise your hands if you know about this one all right good I've got news for half of you um the number one rule for upgrades is don't accidentally reuse storage strats in proxy contracts the consequences can

be so bad so fast that I'm going to be talking about this one first even though to some extent it's common knowledge and it's something that Auditors are good these days about checking and a good auditor checking an upgrade will catch this one so I'm going to move over here and uh use props for just a second testing testing awesome all right so everything that your contract stores goes into numbered storage slots and the variable names in your contract have nothing to do what's actually going on under the hood in the evm so in the evm it's just slot zero slot one slot you know giant hexadecimal number

um and so when you write some code and we're going to call this one Foo here uh the compiler just assigns Foo to slot uh slot zero here and then whenever you set food a three it puts those things in there uh and so now when your code pulls up Foo it reads three um same thing we'll put bar over here and we'll put one thing in bar so if you write a new contract and you change the order or somehow switch things around so that the compiler changes which variable names go to which

storage slots when you read when you read Foo now you get the wrong value not what you were expecting and let's say one of these was your admin address and the other one was your interest rate or something if you switch those two it's really bad you no longer have an admin and your interest rate is now a Bonkers number so that's the first thing you know you can't you've got to read the rules on this one um it's way more complicated than I can actually talk about now but go read the solidity website uh hang on a second yeah so read the solidity website on uh

how those rules work that was the demo there okay so that's the first step is knowing about it second step is check it automatically there's open Zeppelin has some good JavaScript tools that will compare your old contract storage to your new contract storage and see if you've changed the name of anything and so we have that built into our CI pipeline there's also a library called Foundry storage check which I've not used but does try to do a similar thing for The Foundry side but it's important you know put it in so that every PR that goes into your continuous integration gets checked

comparing the old contracts to new contracts and throw a big red mark if you've changed your storage slots so here's a little bonus uh Secret in between the three things uh you want two contrary things when you're doing ongoing smart contract development you want safety and you want speed and these are opposed to each other safety means taking more time uh checking more things and that's the opposite of speed so kind of what separates Great contract development from good contract development is finding ways to cheat this paradox and one of the most powerful cheats I've found is to have security critical code as small as possible if ten percent of your code ensures the

correct behavior of the other ninety percent you can operate much faster when working on the rest of the code and you can check the correct Behavior the Safety Code much easier because it's a much smaller set of code uniswap V2 is a great example of this the protocol split into a safety part tiny pool contracts that only move money and ensure that the amount of money in them just goes up and the rest of the complexity is pushed out into the router that makes the system easy to use but just because the safety checks are in a small place the router can't mess up the safety of the protocol um so this this split between critical and non-critical code is not only a design

great design for building a protocol it's also a really good way to upgrade one because a lot of the contract upgrades can take place in a much less critical manner um and it doesn't have to be in separate contracts it can just be parts of the same contractor's you know code if you want all right so on to the second must-have uh atomic programmatic deploys um sorry so many projects build their deployment plans last after they've written their new code and for some project maybe there's a text file that says how to deploy and eventually maybe something gets put into governance but let's step back and look at the

bigger picture of an upgrade um most projects have two phases for an upgrade first a deployer key creates new contracts possibly configures them and then hands ownership over to a a governance system of some kind like a multi-sig or a much fancier Community Driven governance system and so then that second governance system then actually switches the contracts over from the old system to the new system uh and may do governance config changes as a part of doing that so what you really need to do the right way to do this is to have all of your stuff about the switch defined in one file you

want to uh do your deployer actions do your framework or your governance actions and then if you're running in a local situation simulate that governance action taking place and happening um so this also has a a benefit that you're testing your running deploy your deploy process over and over and over again while even writing the code for your system and so when it happens for real it's not a surprise everything just works and you also want your thing to go into effect in a single Atomic transaction everything happening at once uh otherwise the people get to play with uh an inconsistent state which is pretty

terrible I am going to stick skip that for time reasons and go on to uh kind of the main reason for this uh Fork testing Fork testing is absolutely amazing uh it's the most underutilized tool in smart contract development and I mean imagine how powerful it would be if you could take a snapshot of the entire universe try something new with no consequences and then you know come back with what you learned so Forks allow you to have the effect of making a copy of the entire blockchain deploy contracts and then run transactions as if you were in a copy of the world so there's three valuable things you can do from a fork the first is you can just play with your new code in the Real Environment as if the upgrade had happened and you can run

this you can run a your dap against your new code um or play with it from brownie or something else that lets you just play with stuff there's tremendous value in finding bugs or things you didn't think about you know the stupid bugs in that or just understanding the behavior of the system so you build a gut reaction feel for it secondly you can write tests that run against the forked environment just like your unit test you know you can make sure that somebody can deposit your protocol withdraw from your protocol that a sensible amount of Interest gets earned and it's testing that on your new code before your new code is ever even live and just put it in your CI like everything else and it finds out you know every time you put a push a commit

up it's making sure that you didn't break your existing system um and really they're not any you know they're like one or two percent harder than a unit test to write it's it's really easy to do uh lastly and this is new uh technology these days you can actually fuzz against your fork test um and so this is even powerful you can throw a million randomized tests against your fork environment and see if it works or not doesn't work and what's great about this we heard the question about formal verification well how do you test your interaction with all your dependencies this tests all your interaction with your dependencies and so if you're using Ave like we are or compound or stuff

you get to find out if your code doesn't has some kind of mismatch with their system before it actually goes live uh it's it's kind of amazing um and so Foundry you can then now have uh these built-in uh ability to fuzz onto a fork test um and let's see that's pretty much it this is the conclusion don't be stupid uh I'm on Twitter at Daniel via thank you okay it's time to questions for questions yeah I see one uh can you help us with the microphone

uh thank you for the presentation um might sound as a dumb question not sure but uh so fuzzing seems to uh uh only interact uh with a smart contract on the local uh local environment how do you emulate the adverse adverse environment that exists on mainnet on your local environment is this something that is possible is this something that is lacking or is there a solution so uh let's take Foundry for example uh when it's buzzing it's got a local copy of the evm running and when you're Fork testing you have a local copy of the evm running and when it doesn't know something it goes off

and uh fetches that information that it doesn't know from the blockchain loads it in locally and then uses that from then on um for time reasons I skip dimmuing how Forks work but that's basically a part of how quarks work and so if your fuzzer needs to you know hit the code path that goes talks to Ave it'll go load in the Ave code and then if it hits a code path that uses a particular account it will go load that at users or you know that data from Ave for that account or something and so it basically just works it will make a lot of RPC curls um you know either I guess you know even fuzzing a small amount is still a pretty great idea you know even if you only run 10 000

transactions that's 10 000 times more than you know tests happened on compound for that upgrade um but uh if you really want to honk it you know you'd run your own node to point to locally or you know nearby where you're fuzzing next question okay I saw another question here just to continue for this fuzzing on my net there are a few things maybe can you advise for the best practices the first one in unit test we can really add some predefined computations to check the function works correctly for example

just hard code a few numbers compute them on the paper and so on when you use fuzzing you enforce copy paste so you can go to the sum contract copy this part of the code put to the test so you mean that the function which sum 2 a and b equals to the result of the same function so basically it do not make your system more reliable because you should implement the same algorithm for fasting testing if you want to compare the result which you compute independently with the things you have in the same time and for mainnet another part is very interesting because primarily we

use fasting testing to find like uh problems with huge numbers if you have some overflows and so on when you start on my net you have a pull limit on compound for example 10 Millions it's okay but it means that if you want to provide greater value for the Thousand test there is not enough liquidity and in this case could you suggest to really impersonate the compound for example root role to add more liquidity or what's the best practice to test fuzzing if the existing protocol has very small limits which do not cover the whole

range of possible testing thank you yeah those are good questions um so you kind of hit on this at the beginning but often when you're writing a fuzz test as opposed to a unit test you kind of want to use relative tests you know this account has one eth more than it used to have instead of saying this account has one ease so you're often diffing it uh secondly I'm a pragmatic dwarf and if I can make my life 90 better for a little bit of effort you know I would love to do that uh and so you don't necessarily you know you can do your fuzzing test where they don't have to be absolutely verified to the single thing uh what you're actually going to get as a result you know you if

you catch the big things that's you know a lot of times you know there's Great Value you know if somebody uh ends up with a whole lot of money you know that you know that's a problem or if it just breaks that's a problem um I forgot your last question in there so sorry about that catch me afterwards um I I want to announce the break 15 minutes break next session will start in uh um 11 40. but uh I think that we would have time for one or two extra questions but let's uh say thank you to Daniel and give some noise [Applause]