Best Practices to Deploy Safely On chain by Farhaan Ali

Transcript

all right hey everyone uh my name is fan so I'm the tech lead at Maple Finance uh just a bit about Maple Finance we're institutional focused uh defi borrow lend platform uh trying to bring Capital markets on chain um so today uh so the name of the talk is you know best practices to deploy safely on chain um but actually for me it's more about best practices for onchain Ops so what do I mean by onchain Ops um so the focus tends to really be right now on when whenever we're talking about building smart contracts people talk about the pre-build phase the build phase and testing so you might have seen a lot of these things mentioned before so when you're designing things like what are the requirements for the system you're

building your threat modeling your economic modeling you know when you're building you've got your engineering processes but testing this is where we spend most of our time building smart contracts If we're honest it's like a unit your integration your fuzz your invariance mutation testing if you want to play around with stuff like Gambit your formal verification uh you know red team exercises which are good fun as well when you get together and just try to break your contracts on like an offsite or something uh and then what's the most expensive B usually is your audits uh so internal audit on the team is usually fine um and then you got your external audits uh but I don't want to talk about this stuff today because this has been talked about a lot before what I'm focusing on is what as smart

contract Engineers is our devops basically our onchain Ops so what are we going to talk about so three sections essentially so I'm going to talk about the preparation uh the for deployment the actual deployment process itself so whilst it's for some of you might think okay it's just deploying the smart contracts but it doesn't sometimes it's not as simple as that especially if you've got really complex system so for example at Maple the maple core protocol itself is around 20 smart contracts most of them are upgradeable uh they're all modular so they are not fully dependent on each other so you can upgrade parts of the system uh and that's really good for like a development process of like a product where you might want to change

one part of your system but not the entire system and you've got a live system with you know hundreds of millions of dollars in them how do you safely upgrade and uh develop for this protocol um and then yeah we'll talk a bit about the the post- deployment process as well and what what goes into that so in terms of deployment prep op a so I'm basically talking through like what we do at Maple um so hopefully you guys can take something away from this uh so to start things off we always do Fork testing so this is really easy to do with something like Foundry now uh where you can just like set the RPC um and what we do is we Fork our current protocol and then run all the upgrade that we need to do so you know deploying the contracts doing all the

configuration and then asserting all the state diffs that we expect to happen so you know is the factories pointing to the new contract version have the the contracts have been upgraded pointing to the right implementation uh are all your configuration variables set properly um so that's kind of like a base thing that everyone should be doing in our opinion um then of course deploying to test net is always great especially if you've got like your wider engineering team building the front end the middleware layers like your subgraph and stuff and just uh hooking into your contracts and making sure that everything works appropriately there another thing we like doing is what we call user life cycle testing so the fork test is that I talked about previously is just more about focused um the actual upgrade itself but how do you make sure the

upgrade works as expected so we like to do essentially a complete user Journey from start to finish you know depositing into the protocol a loan being sent out uh yield being generated use it than withdrawing either partially or fully um and you can also fuzz this as well so we've built some utils internally where we kind of basically fuzz random actions against our protocol to do this um and then the thing that takes a lot of time actually depending on how big the upgrade is what we call dry runs uh which I'll get into deeper in a sec but yeah essentially rehearsing against Forks um and another thing you can do is actually PR prepare multisig transactions ahead of time um so I'm hoping most of you guys when you have

upgradeable contracts they're not controlled by a single private key it's usually a multisig this is usually a safe um so one thing you can actually do is so on safe when you go and build a transaction there's this transaction Builder but you can usually just import a Json file which has all the transactions already pre-generated um and this is especially useful if you need to dynamically generate um uh change of addresses or prams as part of like your upgrade process so how do we actually do this um so the deployment process itself we think should be boring and uneventful uh we kind of see it as like a militaristic exercise of like you know you know exactly you're doing ABC so a checklist is super good for us here

so this is an example of a checklist from a recent upgrade that we did um so you can see that usually it's split out into different sections depending on if it's on Main net or like uh another um another L1 or L2 for example uh no Alpha there uh and you can see there's different steps in that so there's like preparation stages the actual transaction uh but then also there's this thing called validation around that as well so uh all right where are we at so the deployment process so yeah we've got the checklist you've dra your deployment so we use Foundry so you use Forge scripts to do this to deploy your contracts uh and what we do is we run a uh the step on on the fork so um we use tenderly for

this so just talking you through how we do this so This assumes that your upgrade isn't Atomic uh which might be difficult to do depending on like how your protocol set up but for us usually an upgrade process isn't fully Atomic we try to make things as atomic as possible but that might be difficult to do in practice um so why do we prefer using tenderly uh so we're not Shilling for tenderly we're heavy power users we've pay them heavily for the every service that we get but they're great um but it's uh using a tendle fork is super good for the non-technical multi- signers as well that you have because I'll show you in a sec there's like a nice simulation that happens you can see exactly all the state diffs uh that are happening so this is this is from um

nosis safe so you can see here just for example we want to pause the protocol you might want to do this before you're doing an upgrade so there's this simulate Button as part of the flow so when you click that you it goes and takes you to tenderly and it runs this simulation now what we do then is there's this button here on the top right it might be small to see but it's called run on Fork uh so we actually when we were doing our big liquidity migration for the V2 upgrade back in what December 23 no 22 um we were moving $300 million of tvl across to the new version of the protocol and we wanted to sim this heavily uh so we actually work with tenderly to make sure that this this was available to us as a feature um so yeah

you can now just run on a fork um and then you get this RPC URL as well associated with the fork so why is this important why not just do this locally using Anvil or something um so the awesome thing here is when you've got the RPC and you've done your transaction against this Fork now you canun run what we call uh validation scripts against this so essentially these are another version of fork tests where you can assert all the state diffs for this specific action that you expect to happen um so that's what I'm talking about at the bottom here with a validation script it's a fork of main net and you can check each state diff and it looks something like this in our case so um just a function called run so you can see here we're upgrading our

Global's contract on eth Main net um so what's hidden here is setup function that all the setup function is just doing is just forking from the latest block um so you can see here we've got logs in here we check if it's like the correct implementation usually there's some helper functions here that are abstracting away some of the more intricate checks that we expect um but this gives us confidence that the system that we've upgraded has been upgraded in the way that we expect it to be um so yeah super useful to do this feel free to check out our repo at Maple core for more examples of this uh and to round things off so just to kind of cover what we've done so youve got this checklist you've run through each step uh you've done it

against the fork each each step before you do your actual transaction on Main it you run your validation script against it so you're confident that your system's been upgraded the way that you expect uh but one thing also that's important to do is to make sure you've deployed the right contract um so this is actually a painful way to actually check before uh cuz you might have had an audit and the audits at a specific commit uh so how do do you know one of your Dev team hasn't just accidentally deployed like the wrong contract on the wrong commit for example so you want to do bite code matching essentially so there's a new feature from Foundry that just came out the I think two weeks ago now um so shout out to The Foundry team thank you for constantly developing for us um but yeah you've got Forge verify B

code now so you can do is you can pass in the address uh and if you do this in your um repo for your contract pass in the contract name as well and it'll do a match essentially for the bike code that's sitting on the RPC that you've provided so if in this case main net and check if it's like a full match or a partial match uh you may want to do a partial match if you've maybe changed the name of your contract and your metadata hash is suddenly changed uh so that's super useful as well so let's yeah definitely don't forget to verify that the right contract was deployed because that happens where people deploy the wrong contract um all right let's talk about post deployment um so you have to verify your contracts of course don't need to get into that too much um

of course having a bug Bounty program is important so there's a lot of work actually happening from the security Alliance so Sam C Sun like from paradigms uh spearheading this uh so there's some cool initiatives there like the Safe Harbor agreement where you can basically sign up to have a process for white hats to recover funds and send them to a specific address and this is all pre-agreed upfront um so yeah hopefully that helps with you know folks wanting to be more white hat than black hat in terms of hacking um and then incident response so always having a Playbook uh not thinking in the moment again you want to just like act and know exactly what you're going to do uh and doing drills for war rooms so uh yeah the security Alliance actually does war room games for you as well so you can reach

out to them and be like hey please run a war game for us pretend hack our protocol and you know make us work um and then lastly uh you everyone needs to do some sort of monitoring for their system so informational monitoring that's pretty simple if you're doing like you know someone's deposited into your protocol maybe you want to slack not ification going in um but I want to dig a bit deeper into how we do critical monitoring so you know how to automate the U up process uh so how we do uh critical monitoring at Maple is we've got a set of system invariants so this helps with our invariant testing that we do um and then what we do is for each invariant we we've basically bunched them up together in a helper contract uh

that then we Deploy on chain and what happens is we ping that contract and check every block if all the invariants are still holding um you may need like different sets of helper contracts so we have one that relies completely on unchain data and then we have uh another that relies on offchain data so what we mean by that is maybe you need to pass in an array of lenders to check like all the total balances add up the way that you expect for example um and then you can set up a Chron job to do this to check so we use tenderly web three actions for this so again tenderly coming in clutch with all their all their products um and as essentially a p pager Duty notification would go if any of the system invariants break uh so yeah no knock on W haven't had one of

those situations yet but you know we do do drills for that and it's always uh very scary when one of those come up um so just to kind of show you how we set this up on tenderly so you can see this is an example of a web 3 action so this is actually live in production right now so this is checking our protocol and variance every block um which is super great and then yeah it'll let us know if something's broken hopefully not uh so yeah that's a my talk on onchain Ops hopefully you guys learned something and took away at least one thing from it um and yeah CH thanks for listening and you can follow me on ZX faran on Twitter if you want to see more about what we're up to thank you [Applause]