How to not get hacked and other security lessons learned | Riyaz Faizullabhoy & Nass Eddequiouaq

Transcript

So today our talk is all about security, lessons learned from seeing incidents in the wild, the amount of attack types, and understanding holistically what are the threats, and then given that, how not to get hacked for your protocol or project or app, and taking all those lessons learned and making it practical. Uh so before that, as I have mentioned, this is us, Nass and Yass. We used to be CISO and CTO at a16z crypto, thinking about security infrastructure kind of across the portfolio, but the broader ecosystem. Um we also worked at Facebook on the Libra project, so thinking about consumer scale security blockchain integration to wallets, custody, and then also have thought about security from the

institutional side. So we're both at Anchorage early. I was a founding engineer, Nass joined also very early, thinking about for institutions, what does security look like with crypto, with holding, but also staking, voting, other web3 interactions. Prior to that, we were at Docker working on security, so more traditional security with OS security authorization authentication and that's where our paths diverge. We've been working together for 7 years, but Nass was on the security team at Apple, is also been working on capture the flag and red teaming cap exercises, and I was doing a a security PhD at UC Berkeley before dropping out and and doing all this in industry.

So agenda for today, really two main focuses. First, we're going to look at security incidents incidents that we've seen in the wild, and really first take a a broad view, just looking across the space, but then deep dive into three in particular with Nomad, Ronin, and Badger DAO. We're going to take the lessons learned from those incidents, but just broadly from what we've seen in the space, and then apply it to be more practical, and think about when you're building a company, product, protocol, at each stage as you think through design, development, testing, what are the best practices, and how to, you know, actually build security in uh for you.

So, we're going to start with overview, and here's a list of some notable hacks uh that we've seen in the space uh by dollar amount. This data is from Rekt uh leaderboard, which you haven't seen it before. Really recommend you take a look. It's actually very very interesting. It shows all the attacks as well as a debrief as well as what was, you know, known to be stolen or mitigated. Uh and as you can see, a lot of these hacks are in the nine-figure range um or all and so, obviously security is is very important. But, I think one thing that we're going to dig into today is if you notice on the very far right is there are different ways uh and different attack vectors uh for each of these incidents. Obviously, key compromise uh being a big one, software bugs if you can think about it from like

a smart contract perspective, but other pieces. And we see website compromise, but really today we want to first kind of center our conversation around understanding that there's more than just the smart contract. And I think everyone in Web3 tends to initially see that as the shiny thing for security and thinking through the smart contract security. But, really there are many classes of attacks, and this is a non-exhaustive uh list of four of them. Uh but, as you can see, there's quite a lot of incidents and just funds stolen across different categories. And some of these categories are Web3 specific. Uh some aren't. Uh and so, we're going to talk about those today. So, for example, a code exploit, that really refers to the smart contract incident. We'll talk about Nomad, and Nass will walk us through that today what happened.

There's been another pair of others uh that, you know, we've we've seen in the space. But, in the next box over with APT, also known as advanced persistent threat, this is a known security attack vector and class for Web2 as well. And it's been a classic where there's an intrusion either by a spear fishing or some other uh compromise. An attacker gets in your system, they start poking around, they figure out what they want to steal, what they want to attack, and then go forward. And so, this is actually, if you remember, way back when the Sony the big Sony hack in Web 2 world, but also we'll talk about them today, but anything Lazarus kind of tends to fall in this category and we expect it to still be a category in Web 3 going forward.

A Web 3 specific class of attack, just given the new decentralized nature of Web 3, is Oracle and governance manipulation. Now that we have the decentralized systems where decisions can be made by, you know, many members in the system in a centralized way, token holders, this actually opens up also a new attack vector where this can be used to abuse the system or introduce decisions into the protocol that may actually hurt it. And so, we you know, we've seen in the space a couple of these. I expect, you know, we expect these two to continue, but again, this is a new to Web 3 type of class just given the new possibilities of technology.

And then lastly today, we're going to talk through an example with Badger DAO, but wanted to bring back a classic also Web 2, but now seeing it in Web 3 class of attack in front-end compromise. Now that we have decentralized applications, we have front-ends that you can interact with on the UI that you connect your wallet to, this is actually now also another part of the critical path for user funds or core user interactions where if an attacker was able to get in there, they can disrupt your protocol or project or app. And so, we've seen this with with Badger DAO and others and we'll talk more today about about Badger DAO. And so, with that, I'll hand off to Nas to talk through Nomad.

Thank you. Um So, I think that Nomad is a very interesting example of security issue due to logic bug. And it's really something that could have been avoided first with better testing, better kind of uh audit kind of like gatekeeping of anything going to prod, but also you know, to a certain extent lack of circuit breaker essentially as part of the protocol. So, let's let's dive into it. Nomad, what is it? It's a it's a trustless uh bridge essentially connecting multiple chains. So, Ethereum mainnet, uh there was Avalanche, and a a few others.

And it really doesn't have a centralized party that is operating the entire protocol. It instead relies on execution of code as part of smart contracts and kind of proof generation and proof verification on chains that it that it connects. Um and so, what happened on on August last year, and I believe that many people here were part of crypto Twitter, right? And kind of like seeing things happen in in in real time. Essentially, we started seeing a tremendous amount of transactions draining the bridge that that that looked very identical.

And so, we didn't see the bridge circuit breaker kind of triggering those watchers, and more people starting to drain the bridge by replicating successful transactions and just switching the the destination address. So, I think that today we should take some time to kind of dive into it and and take time to understand what really happened there. So, just more context on bridges for those who are not familiar. There are kind of like two main buckets of bridges, trusted bridges and trustless bridges. Trusted bridges depend upon essentially centralized entities or systems for to operate and for their operations to conduct, whether you want to verify transactions on one chain

and relay them to to another one and make sure that those things only get executed once. Uh but also trustless bridges that operate without this central party and rely instead on smart contracts and algorithms to to function. And Nomad is is the latter. So, Nomad is a optimistic trustless bridge, that enables secure communication and messaging kind of like relay across multiple chain. So, at a high level, the life cycle and like the normal life cycle of a message across chains looks like the following. A user will initiate a transaction on the origin chain and essentially send it to the bridge

router. The the home contract that lives on this origin chain will essentially pick up this message, organize all the messages into a Merkle tree, and then send them to the corresponding replica contract that lives on the that lives essentially on the destination chain. And after a dispute window, which we're going to explain a little bit, the destination chain verifies and processes the message, and the final step is just execution essentially of the logic that allows this transaction to get processed then for the recipient to receive the message or the transaction.

So, a bit more on optimistic optimistic bridges. So, how does it work? You essentially have the the optimistic verification of messages. And what does it mean? It kind of like falls into three key aspects. The first one is that messages are signed on the origin chain and it actually takes kind of like a timeout period for them to actually be finalized and executed on the second chain. And during that period of time, you actually have an you essentially have like a decentralized set of parties that can inspect and monitor fraudulent activity for the messages that are being passed

across the bridge. So, those kind of like the the entities that decentralized kind of like network of entities that can perform uh you know, verification of fraudulent activity are called the watchers. Those watchers are just auction entities looking into all of the messages that are passing through and can actually um veto those messages if they can prove that this is uh um you know, a fraudulent message. And lastly, uh there is just the trade-off of, you know, uh latency that in general those system those optimistic take more time to get to a finalized state uh because of this um this kind of like time out window.

So, back to the transaction back to the transaction flow. I think that today's going to be interesting to go over a little bit more in depth on how does the replica contract works, which was the the the component that actually um was impacted here. Um then go with an example of the happy path of like us leveraging, you know, the the the Nomad uh bridge whenever we want to to transact. And then what actually went wrong in last August. So, for the replica contract, um you can you can essentially see it as the contract that serves as a as a inbox for messages on the uh the on the destination chain.

It enables kind of the cross-chain communication, uh receives all the inbound messages, and then enforces this dispute like window during which all the watchers can, you know, kind of uh look into all the messages and potentially veto the the fraudulent ones and and flag them. So, that's the replica contract um and in the kind of, you know, the happy path uh say that like Riaz, for some reason, you owe me a little bit of money. Riaz Riaz will want to send $200,000 to NAS. Uh Riaz is on the origin chain. I'm on the destination chain. What's going to happen is that Riaz is going to

obviously send a transaction to the home contract um on the origin chain. The message is then, you know, processed, added to the to the Merkel to the Merkel tree. The root gets updated. An updater will sign an update and then essentially kind of like relay this information to the to the replica contract on the destination chain. And after the time window, the message is proven, processed, and then gets like actually executed and I receive the $200,000. So, that's great. Everything works works well.

And then something actually went wrong last August. So, the Nomad team actually pushed an update to this replica contract. And this update, they actually had done a tremendous amount of auditing prior to this update on the vast majority of the the pretty much kind of like the the the the entire code base at that time started adding a few updates, very benign updates, and then pushing them to production, which is essentially Ethereum mainnet.

Um and it appears that it was not as tested as it should have been. So, what happened there is essentially a vulnerability was introduced as part of this code update where the replica contract allowed unauthorized transactions. So, to go a bit more in depth, the the the update that was actually done was and it was back in April where a default value of zero was actually accidentally set for the new trusted root hash. And this means that actually the bridge would accept any

uninitialized value and consider valid. So, the replica contract process that processes all of those crop chain messages receives uh receives transactions alterations can be made to those and due to the vulnerability the attackers could you know essentially alter various various part of the transaction and actually have it still considered valid by the bridge because at that point in time the trusted route that the bridge was kind of calibrating against was was telling that everything was fine constantly. So, really allowing attackers to claim funds that were meant

for other people. And the the the the hard part here is that this loophole was essentially exploitable by anyone. And we're going to see that this is pretty much what happened. And for those who are here it kind of like looked like the first on chain bank run. Um that was that was everyone could kind of like see live. So, let's let's take kind of like the the happy path that we had before. So, we as wants to send a valid transaction $200,000 on the origin chain the the bridge will essentially kind of like validate the message is valid the valid the the the valid message will be you know kind of like a proof will have

a proof generated for it will be sent to the the the replica contract that is that actually contains a bug now and what happens? Well, it actually works. Just just as we expected because it's a it's a valid message and we actually see NAS receiving $200,000. That's great. But something else happened that day. And what happened is that due to the code logic bug that was introduced in the replica contract and and kind of running with a unintended trusted root hash, attackers could essentially take the the the the transaction that Riaz originally sent for

for NAS, change the address of NAS to their own, and essentially just resubmit this transaction. And the replica contract that runs again with the the mistakenly trusted unauthorized transaction would just approve it. And so, you essentially had I don't know how many people, honestly. I think that it was probably like just hundreds of people are just seeing what was happening and and trying that themselves. A lot of people actually send back the money, but there are very key learnings here, right? Because obviously we're we're all running and building, sorry, systems that will run

in a decentralized fashion. And so, the critical part here is regardless of what you're building, get every single thing that you're shipping to prod thoroughly reviewed, audited. Audits are just a point-in-time view of whatever you asked the third party to review, right? Like they don't have any information of things that are outside of the scope that you give them, and they don't have information about what you push to your repositories after they come in and kind of like have a view, right? So, that's the first thing.

Anything that you push to prod should be reviewed very thoroughly. But on top of like a third party reviewing your code, you really need to have systems in place that enforce kind of like the testing of both the the happy and the unhappy path as if it was production, right? Like here we essentially had something that probably could look benign because it was about state and code of the bridge that were mismatching. So, whenever you want to test your system, and you should do that every single time that you push things, you should make sure that you're copying the state and the code that actually runs in production on mainnet before releasing it. So, extremely

important. And third, finally, ensure that you have circuit breakers that allow you to pause things, give you some time essentially to understand what is happening, figure out the solution, and then obviously get consensus, and then push it to production. So, all those three things are absolutely critical to to put in place. We're obviously going to kind of see how those things kind of uh get incorporated as part of the secure development life cycle, but very important um takeaways. Another very interesting um security issue that is actually that has nothing to do with Web3 and was actually performed by uh some of the most sophisticated

uh attackers in known essentially uh in the world uh was the Ronin uh bridge. So, let's have a let's dive into what what what happened there. So, unlike Nomad, which is a trustless bridge, uh Ronin is a trusted bridge, which means that like it relies essentially on a set of centralized parties to operate the bridge. And more specifically, having a quorum that is made out of uh so, at least five validators need to come to an agreement on which transaction to process as part of the bridge out of the nine.

So, here we have a very traditional kind of like corporate environment. Everyone here is building a company, hiring employees, starting to build kind of infrastructure, right? And the idea here is everyone is going to have some form of access to you know, resources as part of the company, whether it's documents, whether that's systems that are running. And so, here we essentially had an employee that was kind of like on the technical side of on the house, um and then uh and then other entities that are part obviously of running validators as well that are that are independent from from from this party, right? So, very traditional uh very traditional kind of

like corporate environment, but just like the vast majority of people in this room will be at some point a phishing attack was actually performed on one of their employees. And so, what happened there is that this person um got in touch with uh someone else uh via LinkedIn. The person on LinkedIn asked them to uh open essentially a a PDF document on their machine. And the the part that is very interesting is that people often forget that, you know, whether that's SVGs, whether that's PDF, it's not just text or images and and formatting. It's actual code that is running uh every time that you open this file

within the the reader, right? Like whenever you have a PDF reader, it actually will execute uh code that runs within the uh within the PDF. Or not every time, but the vast majority of the time. And so, the part that is like pretty interesting there is if you think about this type of attack, this is something that everyone in this room can essentially be subject to, but also everyone in Web 2, right? Like it has nothing to do with Web 3. Everyone goes on LinkedIn, tries to hire people, and then can potentially open, you know, resumes and and and job offers, you name it right?

So, what happened here is the employee downloads the file. It actually This file actually contains malicious code that is ready to get executed by the PDF reader. And you get kind of like a recipe for a successful fishing attack if somehow the the the the the attacker managed to understand what kind of vulnerability lies in the PDF reader. So, this is exactly what happened. It would actually was a successful execution of the malware, which actually provided to Lazarus. Lazarus, for those who are not aware,

I would say I would call it kind of like the offensive arm of North Korea, of like one of them. Um highly successful, has been essentially performing the same kind of like highly successful fishing attacks in very high profile for decades at this point. And, you know, again, nothing to do with Web3. And they they they know how to do it really well. They know how to gain access to corporate environment and then move laterally within the environment to gain access to more information. They will try to stay in stealth for as long as possible, evaluate all of the resources, and then take advantage of those things. So, this is exactly what

happened here. They gain access, they pivoted laterally to gain access to every single one of the validators internally. But that was not it, right? This is We're up against Lazarus. And so, what they did is not trying to find additional, you know, vulnerabilities within the bridge itself. It actually tried to target vulnerabilities within the entities that have access to the to the to the validators and operate those validators. So, what they did here is after gaining access to four of the validators, they still needed one more.

And so, they ended up pivoted not just within the corporate environment, but actually across corporate environments. So, they actually jumped from one company to the next um through essentially this uh you know, through this malware that was part of that was now infecting all the systems. And so, now they actually managed to find a vulnerability key that was part of an access to another company via this first one, which allowed them to gain access to the fifth validator. And once they had five validators out of the nine, they were able essentially to unlock the access to the Ronin Bridge and essentially start uh creating transactions that would then send them

assets. So, pretty much what happened again, uh extremely motivated, extremely sophisticated um you know, attackers, uh very well-funded, very patient uh that are lurking around in in the space and you know, this is the type of attack that we also need to uh you know, uh stay safe against and really uh protect ourselves uh against. So, not just Web3, really you have to think about security uh on a very holistic way, right? It's really corporate security, operational security, key management, less pri- least privilege uh security, device security. Really, you have to think about it from a all the

angles because this is really how you build a um a secure business, right? Like, it doesn't mean that because you are in Web3, you are more secure. No, you actually have all of the security from Web2 to take care of, plus extra security in in Web3. And so, to actually uh kind of like double down on that, Riaaz is actually going to pre- present another very interesting uh case with with Badger DAO that is again nothing to do with with Web3 specifically. Great. So, let's talk about Badger DAO. So, Badger DAO is a DeFi platform based on bringing Bitcoin to DeFi and Web3 via Ethereum smart contracts. And so, I've oversimplified it, but you can think of

it as a dapp where there's a front end, there are a number of smart contracts that interact with DeFi and Ethereum smart contracts to get yield. And so, to kind of explain more about what happened in this hack, we're going to first talk through the happy path. And so, if you're a user using Badger DAO, at some point sometime down the line, you're going to connect your wallet to the front end dapp. And you're going to at some point submit an approved transaction where you approve Badger DAO smart contracts access to your wallets and therefore can participate in the DeFi interactions that are going to happen on chain with your wallet. So, you'll go to the front end, you'll approve.

That front end will either already have the address or get the address of the smart contracts that you're going to approve with and make sure it's the right ones that are from Badger DAO. They're going to bring it back to you. Say it's like 0x ABC 123, whatever it is. You're going to get that in your wallet, whether it's MetaMask, whatever it is, you're going to sign it, you're going to submit to the blockchain, and you're good to go. You're off to the races, you're making yield at Badger DAO. So, that's the happy path and that's the architecture at a very very high level. But what happened in this incident was we started noticing funds starting to move in a way that wasn't expected and the team started looking into it and this message got posted. And that was essentially describing the root cause of

the attack, which was an API key for Cloudflare, which I don't know if any people are familiar with Cloudflare, they're quite prevalent, used for DDoS protection and you know, CDN management both in Web2 and Web3, pretty pretty common, was compromised. And so, Cloudflare obviously working with the front end of Badger DAO also allowed the attacker an opportunity to inject malicious JavaScript into the front end. And so, when you have the ability to inject malicious JavaScript, what can you do? So, let's see now. So, now you can imagine we have our devil emoji for this front end that is now compromised.

We now have a user going to the same way, going to approve to interact with D5 on Badger DAO. They approve, but now the script can intercept that request. And instead of faithfully going back and getting the right Badger DAO address or getting, you know, what is expected, it can then instead supply the attacker address and just completely forego that flow. And now we're having users approving their funds in their wallets to attacker addresses. And so, the the the the attackers were essentially harvesting these approvals. They also actually very interestingly exposed an increased allowance method. And the reason why this is interesting is that it kind of ties into more about the holistic security picture is MetaMask started getting better at showing very

scary messages of, "Hey, you're approving this contract all of your funds. Are you sure you want to do this?" And so, they did this workaround where they were able to say, "Hey, let's use increased allowance instead," which in these contracts was able to achieve the same result. MetaMask didn't recognize the method, and so it was much more opaque to the users, and they just clicked through and said, "Okay, I'm going to approve this." Essentially the the same effect effects for the attacker. They were able to harvest all these approvals, getting access to these wallets. And when the time was right, they could then, on their own time, go and transfer funds to the addresses that were approved to by those users that were affected by the attack. And so, again, similar to Ronin and we've talked about previously, the lessons learned here is there's more than just

the smart contract security and like thinking about the key management, really thinking critically about what is the critical path about how funds can move or how an important interaction in my system goes through and thinking through all of the dependencies and core components along the way. And these dependencies, they may not even be your own software, right? It could be Cloudflare, it could be some dependency that you vendored into your code that is maintained by another another company, another open source library. But really understanding what is the security bar across all of these components, what can go wrong, what do I need to plan for in in case of, you know, compromise of an API key, how do I detect that, and going forward from there. And in a similar note, I'll add on to in the in the class of front end attacks, we've also also

seen attacks on DNS systems, so you can point to a different front end, a different website entirely, attacks on name servers as well kind of in a similar vein. And so this is with dapps, you know, having a front end, even if you have key somewhere else, this is still a critical part part of the system. And so with that, I'll hand off to NASA that we've kind of seen an overview of and deep dive into these attacks, some lessons we can learn and practical advice. So now, I think it's very important to take quite a bit of time to go over, you know, not just the the learnings, but how to make kind of like actionable kind of like take actionable kind of learnings from from those things. So, first and foremost, we talked through

several examples, right? Like there is kind of code bug like, you know, within the within kind of like smart contract level, contract that happens to be non-upgradable which prevents, you know, any form of solution to be put in place if not already present. And we've seen also kind of like, you know, traditional web two attacks with whether that's fishing, whether that's taking advantage of a weaker link in the in the supply chain.

And so really all of those things make make it clear that, you know, we need to tackle security from all angles, right? Like this is the bar that we need to get to to keep the company safe, the products the the the product safe, and the users safe, right? So, security really should not be an afterthought for anyone in this room and you're going to iterate more and more over the product. So doing kind of like taking care of security early really compounds all the time, right? If you think about it, it's like you everyone here is going to keep building great things, right? And you're going to iterate over the the product and just get more things out. And as soon as you kind of like get systematized kind of

like security as part of, you know, the supply chain and kind of like your automation systems, you actually will have that in place for every single iteration along the way. So really let's make security scale and this is what we're going to talk about here starting with kind of the the software development life cycle and how to introduce security into each one of those steps in a way that is very actionable. So first and foremost we're seeing kind of like, you know, boxes here.

The real version of of the the the software development life cycle if you want to think about it is really a loop, right? Like the idea is that it's not really just a set of iteration, it's pretty a set of iterations over time, right? That kind of like starts with designing, then testing of your software, deploying it, getting it in the hands of your support in in the hands of your users and then monitoring and kind of like having visibility into whatever is is happening. So that's really designed to help you enforce the best practices over time across product iterations.

So the the the software development life cycle as many of you know starts with the design phase, right? Like the design phase is the the moment where developers essentially detail the blueprint of the architecture, the different services, the different components, the the modules, the interfaces, uh data models, you name it, right? Like this is really how the moment where we all think about what product do we get in the hands of our customers and and users. And really this stage ensures that the system meets not just the requirements from a functional perspective, but also establish guidelines, best practices,

right? Like whether we're talking about just good quality or just security, many things, maintainability, you name it. So really important to have that in place and as part of this comes security, right? And one thing that people kind of like overlook, but it's actually extremely important is using the right tool for the job. Like anything that you can do to bring simplicity to your day-to-day is actually more time that you allocate for thinking about security, to think about just adding better features. Really complexity is your enemy both from a functional perspective, but especially in in in security.

Second, the thing that we will hear, you know, the most extremely often in in in this space especially is to never roll out your own cryptography. So I know that there are some people in in the room might be building on top of pretty advanced, you know, cryptographic building blocks or developing some themselves. If you don't have a choice, obviously this is the path that you want to take and but you want to do it the right way. There is no kind of no one here should ever, you know, kind of like cut corners. If you can avoid to roll on your own crypto, just do it.

There are tons of things out there that have been kind of reviewed by the entire cryptography community that are ready to be used by by everyone essentially in the space. And finally here, uh we get to uh to kind of like a the cheat sheet is really understanding your tools and and dependencies and the fact that each one of those things are introducing risk, right? Like anything that you leverage from a third party that is outside of what you've built is also part of the attack surface that the attackers have access to to take advantage of your systems, your product, your own users, and all those things need to be part of how you

think about the the the the the attack surface of your own um you know, of of what you're building essentially. So, a quick cheat sheet uh for code development and and of course we'll we'll provide the the the the slides after. Not going to go on on every single one of them, but probably like two very important ones that like we've seen uh a lot of issues around especially recently. Uh the let's I think that like the the the first one is probably dependencies and kind of like uh dependency security. So, you know, again, every new dependency is a risk and to limit it, first obviously review thoroughly every single piece of software that you use as if it was your own, right? Because the attackers, they actually it it

doesn't matter for them if it was written by someone else or or your own team. So, they will be able to leverage it either way. So, really review it as if it was your own code, and then once you have a proper review in place, hardcode them. Ensure that it's the right hash uh like you have like the hash immutable hash of like anything that you're running, that you're verifying, uh signatures if possible, or actually even import all of your dependencies within your actual code repository to ensure that your code it lives alongside its dependencies.

And second, circuit breakers. So, we talked about kind of, you know, circuit breakers not being present for example for the for the Nomad hack and that once the issue was found there was no way essentially to to prevent anything from happening. It's even more critical when we're talking about web three. So, to think to to think about circuit breakers the right way, I think that you have to position yourself in as if, you know, the attacker already has, you know, researched issues in your own software. They taken time to test it thoroughly whether that's like on a testnet, on a fork of mainnet and kind of like really they're ready essentially with everything in place to take

advantage of your systems or people. And so, your time here is the time actually works against you in this in this environment, right? Like your goal is to actually give yourself kind of breathing room and really just expand the time window that you have to understand what's happening, what kind of solution can you can you build, and then how to deploy it and make it accessible, right? So, really having those things in place is extremely important and it's often time about time. So, and the the additional thing that I would add is most hacks in this space actually could be prevented by adding restrictions and constraints on the kind of like 0.1% of the scenarios, right? Like trying to move an amount of

money that is, you know, obviously extremely suspicious anyway and would only impact kind of like the top, you know, like 0.1% of the users. And so, it's it's very important for people here to think about trade-off whenever you know, designing circuit breakers because it's often time fine to, you know, kind of like bother those kind of like 0.1% users for the rest of the the, you know, the people using the the the protocol to be safe. So, think about, you know, when it comes to like with we think about pause functions for your contract. Think about obviously upgradeable contracts that allow you kind of like pushing fixes to, you know, to the network.

Whether that's baking periods or, you know, governance kind of governance proposals to to to take to take effect. Time locks for sensitive actions on on contracts. So, like all these things it's kind of like layers in depth that kind of like stack and make the hard the the job harder and harder for for for the bad guys. All right. So, now that we have code, let's talk about testing and pre-production. So, code is written, it's we're on the path to production, and we've already discussed this, but testing is your friend. Especially for security, especially for systems where the core part of the system involves financial tokens or other monetary value

where correctness is so intertwined with security. That if something goes wrong and is not correct, it means there's a there's a security vulnerability in your system. So, definitely definitely have a robust testing plan beyond just unit testing code coverage, really on integration testing with both code and state. Like we were talking about with Nomad. Thinking about it very holistically, how you're testing, if it's is it actually mimicking what's in production. Is it actually realistic both for the happy path and then thinking about the unhappy path and how to stress test your system. Um on that note, on the testing and like this environment, best practice is to treat staging like a production environment. And for a few reasons. One is if it's a mainnet fork or as close as you get to mainnet, get the conditions and the state as close as possible.

Two is when it comes to things like responding to security incidents, to getting paged for, you know, something going wrong with, you know, capacity, whatever it is. You're training yourself and getting yourself ready for production. So, that when it happens in production, it's already stressful enough, but you have a playbook, you have a runbook, you have a checklist of how you address issues and it's been battle tested in this environment. And this environment, by treating it that way, by being paged or being, you know, alerted for staging and treating it like that way, you're getting that battle tested experience uh for production. And so, you're ready to go when it's when it's time for prod.

When we are setting up the infrastructure both in staging and production, this is a good time to think about how we are going to lock it down. And two classic security principles here. So, one is principle of least privilege. Uh idea here is not to give any more permission than absolutely needed to any person or system. So, for example, if you have a let's call it a service that has the ability to add a user. Probably shouldn't also give it access to manage smart contract deploys or some other sensitive action. And so, understanding what are the security boundaries between different components of my system, whether they're smart contracts, whether it's systems, understanding the security level of each and the privilege level of each. And on similar note, other security classic is separation of duty.

And that is, similar to the above, making sure a single entity doesn't have the ability to do everything or too much. And so, this often also culminates in, you know, sensitive actions like contract deploys or key management. There are a number of best practices here like using multi-sigs, you know, Gnosis and others. And so, there are ways to like split this up. But just just knowing where these high privilege systems are and understanding how much you can pare down and how much any single system has access uh in your environment is very important. And of course, we can't talk about getting to production without talking about audits. And as we were discussing, think of audits as you know, they're a piece of your security story. They're not everything in your security story.

They're a point in time for a certain scope with a certain auditor who has a certain set of skills and expertise. And that's what you get out of an audit. Still very valuable and you should definitely do it for whatever you bring to production, but do it with that that framing and understanding what guarantees you're getting from the audit and it's not your full security story. The thing with the audit, too, that I think is very valuable to as you get on doing audits, is understanding the best way to maximize what you get out of the audit. And a lot of this is like what you put in is what you get out. And what I mean by that is being very involved in the audit. Often with audit companies, you're able to ask for background on who they have on the team, who can work with

you, understanding is a fit in terms of like their their background, if they're like very good at solidity or some other like some other technology, if it matches with you or not, and you can be selective in in that case, and it's good to be. Also, making sure that they're on board and they're ready to go. So that when they show up, it's very clear to them how to best get into the code, how to understand what they're even looking at, understand what their scope is to do, and they don't get lost in rabbit holes. And so, often, you know, when I've done audits with NAS, we are very hands-on with the auditor, daily check-ins, or whatever makes sense. Just really make sure the things are on track and you're not straying away from what you're really trying to get out of the audit, and you're getting things moving

forward. Other thing too that I've seen as a pattern and I think is worth discussing is that it's okay to have multiple auditors. Like you can't expect one person to be able to understand everything in your system, it's front end, it's smart contract, back end, and it's totally okay to have multiple auditors who are specialized and are experts in those fields. You may have like one auditor who's doing your AWS infrastructure, another audit firm or auditor doing smart contracts, and that's totally okay. It's a little more overhead, but you're able to really maximize the value of what that person is an expert in and can give you that best advice and best review. And so, you're not getting kind of like best here, but then mediocre on other parts of the system.

And so, that's on the audit, definitely take the time to really put in effort and so you can maximize what you get out of the audit and get a really good audit. Okay, so now we have set up staging, we've done an audit, it's time to go go to production, super exciting. And really key takeaway here is you should be able to know as soon as you press that button or whatever you're doing to get production, you you put your contract, you put your your infrastructure, you should very quickly be able to understand, or even before you even push the button, is what's actually running there. And what I mean by that is are you able to point to an exact get hash, an exact build, an exact chain of commits or pull requests or reviews, or exactly what was audited, so you have the confidence that you know what's in

there? And by doing that, you know that if there's an issue, you already have the context of what is the exact version of what I'm doing and what I have deployed. What were the changes from the last time? I have the understanding so that if there's an issue, I'm able to then very quickly diagnose, or at least much quicker than if I didn't know. And best friends again tools here that you can use are reproducible builds, so make a build always spits out the same code no matter or same uh builds no matter what, you know, as as whenever you build at different times, having the same shot, you know, documenting, versioning, tagging, making sure you're organized around here is really helpful.

And then on that too, having a checklist uh both for happy path, understanding what's going in in terms of both code and state and builds, uh but then also if you need to make a hot fix, if you need to do a rollback, what does that process look like? Having that runbook and checklist defined and testing it in staging uh in the past phase will make sure that you're ready for production. If you deploy something and things are not looking right, you can very quickly jump in and diagnose. And so you should be able to kind of pre-deployment, post-deployment, have these checklists, have this understanding, and it'll help you both for just functionality, but also for security, right? If you have an issue, you're very easily able to understand what what did I change, what do I need to look at, and get that context.

So now we're in production. And now that, you know, that's not the end. Uh we have to actually support what we have in production, making sure that it's a good experience for our product, our users, our protocol. And so here, making sure that you have monitoring, logging, and alerting. And what I mean by that is for monitoring, you often get kind of the the base like CPU usage, memory usage for free from cloud providers or other systems, but understanding and having a picture of what's happening on on my machines, do I have custom, you know, emitted events for my smart contract that I'm able to look at and aggregate and see what's, you know, are there a bunch of users on boarding the platforms, everything looking normal, being able to understand that, having that data is really important both from like a product perspective, but then we'll see from

security. Logging, understanding both happy path, unhappy path, business logic specific understanding of exactly what code paths are triggering, when, how often, um and being able to basically have all that data because you need both these inputs to really have a strong incident response and alerting program. And what I mean by that is that you should be able to take these events, whether it's through monitoring, stats, or logging, and be able to trigger alerts that your team has, again, a checklist or a run book on how to under- how to triage, take in these these alerts, how to start with diagnosis process, if there's anything in terms of just, you know, security and how to think about that, but then a tight response loop of we see these we see these logs, we see these

monitors, we see these alerts, do we need to then roll back, do the circuit breaker, take any action, roll forward, but having that in place that you're not caught blind and you're not finding out of, you know, however many days, hours, weeks later that you've been potentially hacked and then having to figure that out post facto. And so, really important that you have this continued support even though you've pushed to prod and life is great, it the story doesn't end there. So, we kind of like saw the the the the four kind of like main steps along the way, um but that's not it, right? Like as we mentioned, this process is not going to happen once, it's not going to happen twice, it's going to happen

many times. Everyone here is going to go on to build amazing things. And so, you need to make sure that you keep designing and shipping great features safely, right? Like this is very much kind of like a an iteration. And this framework is really here to help you kind of think about security throughout all the steps, right? But also throughout all the multiple iterations of of the company. So, we're back to zero every single time for every single iteration, and that's great, right?

Because that really means that your system will actually apply the same verifications over the multiple like layers of the defense in depth for everything that you push, everything that you design, everything that you test, kind of like always having those things as systemized as possible. So, yeah, the the last word here, we really want, you know, everyone here to kind of keep in mind quite a few things and and and take aways, uh, you know, that you get to to apply as early as today or yesterday already, um, is so, first and foremost, security threats are not just bugs in the code,

right? Like it's it's really it comes from all the angles. We're we talked about, you know, corporate security, operational security, uh, product security, front end security, back end security, uh, product key management, you name it, right? Like there are all of these things that you uh, you have to be extremely intentional and conscious about whenever you think about building them and making those decisions. But also kind of, you know, checks and balances to ensure that all those things are done securely. So, therefore, make sure that you think about security holistically, right?

Rather than like a checklist item. The worst thing that you can do it is thinking about the security like, boom, I check the box, like I can essentially move on to to the next thing. No, it needs to be like very intentional and and thought through from from the get-go. So, as part of that, automation, as much as you can automate and just like add as part of, uh, you know, well-defined processes uh, that you go through every single time with the team or like let your systems do that. It's very important to kind of like keep that do the work so that you can go on and focus on on building other things.

And of course, you know, we talked about secure design, we talked about we see a lot of unit testing, we see we I think that like you integration testing was probably like that the thing that we saw lacking the most in general whenever like looking at systems. So really make sure that you have a system that really test things out as if it was production with the same code, the same state, everything the same and treat it just like it was in production, right? That's really that the bar that you should have for for you know, essentially building in in web3.

And so this is this is really a marathon, right? Like that that's not a sprint. You'll get to really add things over time but just start thinking about it from, you know, today. This is the right moment to really be extremely conscious about security and really have it scale within your product, within your corporate environment, but also your culture and your DNA as a company. So, thank you so much and very excited to to answer any question. Hey there. Thank you both for the talk, really incredible. Two questions. First one is on the topic of circuit breakers.

You touched on a lot of different functionings that are pretty important so pause functions, things like that. What's your take on how to balance that with decentralization and not like going back on commitments to users about software that you've deployed because of course these are all like processes that you may need to change code or deploy new contracts for. Yeah, I think that like on on the ones that we mentioned, you know, when you think about kind of whether that's time locks, whether that's you know, kind of like governance kind of like proposals, you know, having a baking time, all of those things work really well hand in hand with essentially kind of like decentralized systems. I would say that probably kind of like the the pausing is probably the thing that is that is uh something that you need to think about from kind of like a

automated perspective. Like if you can automate, you know, kind of like the pausing based on several factors, whether that's, you know, too many transactions, you know, like just a two too much volume, you know, like flowing through like a per unit of time, you know, you kind of like define those things. But I'm pretty sure that in general the community will be very excited about like making trade-offs there. So long as it keeps things like as secure as possible and you know, you prove like really well that you cannot DDoS essentially the protocol through, you know, kind of like malfunctioning of of those I did multi-sig here is your friend. So like you decide like what is most sensitive versus what can be automated and but multi-sig is a way for you to like say

we're still decentralized to like this degree for like as if, you know, same with like the client contract. So there's no one like size fits all and how you set that up or how what is multi-sig versus not, but you can think through like what's going to be automated versus what's multi-sig and how big the multi-sig should be both for like response time trading off the security. Got it. Second question really quick. As people that live and breathe security and have been around this for a while, of course if you get burned by any of these hacks or things like you care about security. Like if you're just a company, even if you have nothing to do if like security is your main focus, but how do you guys sort of like pitch all of these things to audiences that maybe don't have this as priority number one

cuz a lot of this is like if you get burned retroactively you're going to care about it. So if you've never been burned Well, I feel like we've done the the full radio on this. It's like I think that at the end of the day you need to make security core part of the proposal of the value prop, right? Like security should be a feature of the product rather than the kind of like the checkbox. Like if you can advertise it, leverage it and kind of like, you know, a lot of people are going to Bitcoin because like among other things it's also like extremely secure, having like extremely thoroughly reviewed. And so, and and many other things that that that were done as part of the, you know, development of Bitcoin. And so, I do think that to a certain ex- to to a certain extent, it's

really on people in this team to understand how to make the security narrative work for them, right? Like, you're going to do it probably anyway, so you may as well just make it work towards kind of like the value proposition of what it the whatever it is that you're building. Yeah. I think there's also like there's going to have external pressure anyway, like whether it's community or like regulators, that that's one piece. And then there's the internal piece of like beyond that, what is your security culture and what you think makes sense at different stages. And the way that we often think about it, too, and we've done from doing a lot of custody is thinking about security as economics.

And like, what how much money would it take to hack our system? And how much money then should we invest in security? And thinking about that's like another framework you can use to just think through how because you're not going to go full overboard like we're going to put everything in like the Swiss Alps for like this thing like at at startup stage. But thinking through that and thinking through the path as like and being one step ahead of attackers is like as you grow, okay, we're getting on like we're getting bigger. Like it's we need to up our security game because this is what we've like thought through of them as a model. Yeah. I think it's like the the unique challenge here is probably kind of it's very easy to kind of like see, you know, the the chart of like whatever value

your product and your company is protecting of like quickly rise, right? Like, so this is why like the piece of like staying one step ahead is always critical cuz you should still kind of like hedge on like anticipating such growth essentially of kind of like the value that you're protecting. I have a question around these some of these phishing attack cases like the Ronin Bridge. So, that attack was initiated through phishing, and I'm wondering what are your recommendations as far as some preventative measures for phishing attacks? Is it better employee training and compliance? Is it anti uh malware software on computers? What do you think is a good way to prevent these types of attacks?

it's a good question. I think there's The way I think about it is just think about the layers and defense in depth. And so there's good there's practical steps all along all the way you just mentioned. So like from from the get-go like from your email or other point services, what kind of scanning do you have in place? What kind of protections do you have in place? So So for example, like Gmail has like advanced fishing like scanning. There's other companies that you can add That's one touch point. I think the other broader touch point is you should model the system or at least we would in the kind of a security custody system as well is like think about like what happens if kind of change the framing of the of a fishing is like what would happen if your laptop was open or your employee's laptop was

open in the office, someone grabbed it. And like they would they had the laptop, like what could they do or what could they not do with the laptop alone? And thinking of it as like okay, imagine we're already hacked. What's possible or not possible? And what for what is possible, are there things you can put in place whether it's additional layers of control with like multi-sigs or like other ways we architect the system to limit the blast radius of that from happening. So often too like in a common audit like security audit test will be you give the auditor employee laptop and say have fun, like tell me what you can find.

Um but that's that's another way to just think about it and like beyond like the initial layer defense, what is what do you have behind? Uh the point that you made on basically Web3 having these expanded protections and essentially being even harder to secure compared to Web2, I think that was really salient. Um given that fact, I have kind of like just kind of two questions. One is the biggest companies in Web3, uh so your OpenC's, your UniSwaps, um kind of how far are they on the curve for adoption of kind of enterprise security solutions? Like are they already using things like Octa, um mobile device management and things like that? And then kind of the second part of the question is like it's basically

what are the like design choices that different design choices that they these Web3 companies have to make given that it is like an increased security risk compared to Web2. Yeah, that's a that's a great question. Uh so we actually work pretty closely with uh Corey and and the team at at OpenSea for example and and other companies. Uh not to kind of like reveal too much about kind of, you know, the the state of things, but it's you know, the the the Web3 companies understand really well and we have kind of like we we used to have kind of like constant conversation with them to just like ensure that uh you know, the path between again kind of like the the the economics of like what they protect and you know, and the

responsibilities and like what they have in place always kind of like follows the same line, right? On on the graph. And so uh we've, you know, a lot of work is being done by the teams obviously kind of like everything in place rather that uh you know, device management, whether that's like uh intrusion detection, kind of like already uh incident response kind of like um you know, relationship already established with like, you know, the best companies in the space so that like if anything happens getting already being ready, right? Because as I mentioned, I think that like that that's something that also people here should should take into account as kind of like the the the growth kicks in, make sure that you already have a relationship established with an incident response

team because like those incident response companies uh it takes a lot of time if you don't have like the right relationship with like, you know, the CISO, like the the the right people internally, it actually takes time to get in touch with them, establish the contract, and kind of like get moving on the attack. And again, as I mentioned, the the time is working against you, right? Like you don't have time to like send an email, a LinkedIn message, wait for the person to enter none of that, right? So, uh trying to kind of like get ahead and already have those uh relationship established early on is pretty important. So, just wanted to to put that out there because that that's something that kind of came up pretty often. Um And so, yeah, we're we're actually I think that there's a spectrum,

obviously. I think that it's fairly hard often times for for companies in Web3 to understand at which level of the journey which which step of the journey they are. And I think that this is uh this is pro- this is just good to have like those conversations so that like you understand that it's actually generally a lot earlier than what it seems, right? So, but overall like I have seen a great people building, you know, great conflict security features within the entire A16Z portfolio, for sure. I wanted to ask about testing and kind of like, you know, since college CS classes, professors would be like, "Test your code, write your test cases." And people would don't listen. And like now, uh with developers like, you know,

there's like a trade-off between like creativity and like putting them under pressure and like you have to like write these codes. So, just thinking about like, you know, that like what's best practices to ensure, you know, your engineers are on it and like are you still using it as a creative process? Like is automation the key here? What's your guys' thoughts there? Good question. I think there's I think part of it is cultural and setting the tone of we expect it you know, every you put up a pull request like this is the bar of we accept as a company or whatever. And like and that bar you set a foundation and you can have that scale without you having to police and kind of like checking on it and like you know, whether that's like having always mandatory code like code reviews by certain code owners, having a certain

code coverage is kind of one of those things that like don't it doesn't tell the full story cuz you can just write not so great tests and still have great coverage, but like there's other kind of ways to try to encode that. But certainly making and thinking of security as part of that that build like build like development you know, to testing as part of the process and just embedding it as much as you can is a way to just get things started and then over time you can add more as you know, as you grow and like in that in that case, but certainly I mean I personally advocate for always having testing along with a feature cuz like you know, does it work if and like both from like a correctness but also when you're dealing with tokens or other value, it becomes a security concern as

well. Yeah. By the way, like on the code coverage, I think that like a lot of people often times like people just think about it from like a code coverage during like unit testing. Make sure that you also add that for like integration testing, right? Like it's not just you also want to make sure that like your integration testings like really cover a lot of the path that your your code takes. And like more broadly, yeah, like really fully agree. I think that like it's often times just like more cultural. I do have to agree that the kind of uh software infrastructure and kind of like tooling that is available like within web3 on the integration testing side is probably like a lot weaker than like where it should be. So I think that like that's like as a as a space, something that like we need to to do a

much better job at. But like there is there is still kind of like a lot being done there and and really the the the you know, kind of like the culture piece is like doesn't really rely on kind of like front end work that like you might be doing back end like anything really it is really just all-encompassing and really will scale well across the entire company as you grow, as you go in different directions and kind of like build different components. Um I also have like a thousand questions but I'll try to keep it to just a couple. Um one is like you know, you mentioned how you want to make sure only like audited code makes it on to mainnet and I've been trying to talk to a bunch of teams and it seems like there's not like a consistent way that people are tracking like what's like

what are their devs deploying? What like there's like a lot of different process around it. Like have you seen any like useful tools or approaches people have used to one know what all the code they're like what all the contracts their dev team is deploying to you know to production? And also is there any tooling that helps with like tracking what's been audited, what hasn't, and making sure only audited code makes it there? Uh that's a that's a great question. I think that like so far I I I like in the space I've mostly seen that being manual. Kind of like just you know, kind of all the contracts might be referenced and like you know well, whether that's a unified like other platforms they kind of like just like lays down like hey, here are like all the things that you know, you can

kind of like go ahead and and have a bug bounty on or like just all the things that have been reviewed. I haven't seen kind of like something that really kind of like almost like source graph style just like gives you like this thing was part of the code and like this thing has changed every since then. I think that like I haven't seen that um happen uh or or being built. Um so that's that's uh that would help certainly especially if like yeah, like just a decentralized like communities to like look at it and just like very understand like very quickly understand like okay, that was part of it. You can kind of like limit even from like a financial perspective, right? Like you you don't want to like go over like the same thing again and again necessarily if like the entire component doesn't like didn't

change. But there might be obviously like side effects which is like one of the reasons why like often times people tend to just like re-review everything, right? There are like interactions that you may not have thought about. For example, like the no the no meta hack was like an interesting one, right? Like the vast majority of the like the vast majority of the protocol didn't change. Just like the the the code that interpret some like already existing state like changed. And so like this actually created the the the issue. So I wouldn't say that it's because like you already have something that tells you like hey, this is everything that has been reviewed that like you should necessarily have too much confidence about it because like of these kind of like side effects.

Thank you. I just one more question is like around the monitoring um have you seen, you know, how act you know, there's people are making like possible contracts or other ways to kind of like put the brakes on when they like manually detect something is wrong or they find out something is wrong, but have you seen uh you know, from some companies you talked to, they seem a little bit cautious on like making so that like say like monitoring detects an anomaly and then it can maybe like pause their contract or something like that. Have you What about you mean the automation? So essentially Yeah, like automated system kind of like pause the Yeah. Yeah. Cuz I I we were looking at that stuff like Forta and others and I think one challenge they've been having is um you know, they have all these alerts

and they know like they detected like an hour ago that something's been hacked, but it seems companies are somewhat cautious around like, you know, taking that and then actually connecting it into the contract. Have you seen anybody want like actually implement like where monitoring can pause their contract? And like what do you Do you think that's like something that can even happen in the future? Just like that there's kind of like the liability piece that is also like tricky. Yeah, I think it's there's like kind of like a also from just a depends on the protocol, too, like we talked about the decentrals like like the how decentralized is it versus automated. There's no one-size-fits-all here.

I'm off top of my head, I don't know if I have any examples of who's been automating it, but it's certainly like in my mind, too, there's like there's the pause and then there's also the resume and there's ways to play with this one takes can be automated, this one has like a multi-sig how big is the multi-sig? There's a lot of variables there, and so I think there's not going to be a one-size-fits-all or haven't actually seen a pattern yet, uh but I think it's it's interesting question and worth watching. Yeah. And it's mostly also because like there there is no one size I think it's like if you want to pause, that means that like you need to expose essentially the the you know, kind of like the mechanism, right? And kind of like have some form of agreement and consensus around like who should be able to like

call this thing. And I don't think that like a lot of companies will would want to kind of like carry the the liability there. I mean, even if even if they they don't have access to like the resume kind of like piece, it's still kind of like a liability. I'm not too sure. I don't think that I I've seen that happen. Um Yeah. The other the other piece that is hard there is that um you know, often times like a lot of companies like understand that the a lot of the attacks are kind of like single transaction, right? And so sometimes it's it's just like hard in nature to kind of like detect things uh because it's not like you're going to have like many opportunities to kind of like learn what the attack is going to

look like. So, yeah, I don't know. All right. Awesome. Well, thank you so much, Nassim Aoued. Appreciate it.