Crypto Built the Rails: Next Up Operating Them Safely

Transcript

[applause] >> Hello everyone. Uh so I want to start this uh presentation with an example. So a couple of months ago um there was a protocol team that was doing something what you would consider pretty standard. Um it was a regular business as usual upgrade of their protocol and nobody expected anything to go wrong. It was a classic classic Tuesday.

So the test suites passed, the contract audit was fine. The governance passed. The time lock, also good. So it was all systems go um go and deploy and carry on with our day. But something happened. The moment the the deployment hit the mainnet, something strange started happening. An asset which was supposed to be valued at $2200 uh immediately showed $1.

So this started a terrible chain of events. Uh bots immediately picked it up. Uh the cascading liquidations came into effect. And the protocol within 4 minutes uh ended with almost 2 million of bad debt. But the worst part it wasn't the smart contract that caused the issue. The smart contracts behaved as exactly as they should, exactly as intended. Uh all of the tests didn't show anything that um it could have been the issue.

So what happened here? Let's zoom out. So as Kartik mentioned, uh my name is Andre. I'm the co-founder and CEO of Tenderly, and I've spent the last 8 years building the simulation layer that some of the best teams in the space have used to stay safe on chain and build the best versions of their products possible. I want to talk about one [snorts] often overlooked aspect of running a blockchain product which is important now more than ever which if we want to if we want to get to the next level of of adoption.

So, looking back at our example why did why none of the checks triggered any red flags and stopped the process all together? It seems like a very egregious mistake. It is something that should have been that should have been picked up. But the reality is often times most of our energy, most of our systems, most of our practices all point into the direction of getting to mainnet, preparing the smart contracts from just an idea to something to something concrete, something that launches, something that everyone rallies together.

Now, this is definitely the right place to put your attention at the very beginning. It is a serious milestone. Nothing can exist without that without that core step. And we need to ensure that all of the audits, all of the tests, all of the checks pass before we hit launch. But we often hyper-fixate on that too much. It is important to get it right, but it's only the half of the story. At some point, you launch, you hit mainnet, and the work in the building phase is done.

But the work on operating it and making sure that it continues running smoothly and continues getting the upgrades and the attention that it deserves, it never stops. So how did we get into this situation where we spent the last 10 years looking at how do we clear this build stage while often overlooking the essential piece of operating this protocol in production with live state, live dependencies, and that entire world.

There's a saying that I that I like that uh paints a bit more of a clear picture. Uh the map is not the territory. Now, this is a sociological uh quote, and it basically says that uh if you're looking at a city and a map of of a particular city, uh this is a clean, concise, documented instrument. A map is on paper. It's very precise. It's very proportional. And it gets to be very accurate um mapping the city that it's trying to represent.

But, this is not uh the entirety of the story if your job is to navigate through that city and live with it day-to-day. And this is the territory. So, on top of the map, if you start adding the traffic, the congestions, the construction, population density, a city is a lot more than just a map. Um the territory in which it lives is just as important in knowing as knowing the the layout of the land. And I think crypto and blockchain is a lot a lot more similar than that uh to that than than you might initially think.

We spend a lot of time defining these smart contracts, making sure that they are um fully audited, fully tested, as tight as they can possibly be so that they can lock in the essence of what we're trying to achieve on chain. But, this is only half of the equation. When we start uh when when we deploy it on the mainnet, um it starts getting a life of its own. All of the dependencies, all of the all of the assets, all of the user flows going through this um uh contract, it's a life of its own, and it is um completely different to what we initially uh to what we initially expect.

So, in a way, in crypto, you build a map, but then you need to operate in the territory that's broader than it. We've been able to survive this way of doing things in the space for for so long because hey, it is a new, innovative, disruptive technology. Uh we sort of needed to take on the early risk, or maybe even needed to take the early risk if we wanted to get to where we are at today. But the situation is different now.

The institutions are very clearly showing signs that they see crypto as the future rails of their industries, and uh we are we are no longer considered the alternative finance. We're starting to become the finance itself. So, we need to start showing up on the same level as it's expected of us if we want to meet a demand uh and achieve sort of full scope of what crypto can do. So, how we manage risk is going to be the essential discipline for the next decade of work, and how we protect

>> [snorts] >> and uh and um guard and grow uh the institutional assets, interest, and the and the corresponding audiences is going to be critical if you want to succeed long term. So, I want to talk about four types of risk uh that I see as sort of a complete unit in when it comes to operating um a pro- up- on-chain protocol at scale. It's process, it's dependency, it's market, and code. We're going to walk through through them one by one.

So, the first type of risk is the one from our example. This wasn't an issue with the code. Uh it wasn't an issue with uh anything that um was malicious. It is a process or an operational mishap that uh caused a severe uh severe layered uh negative consequences. And how I think about this is that every um every process or every rule or every method in which your organization interacts with the blockchain is another opportunity where things can go wrong.

So, signing regular transactions, uh running treasury management operations, uh deploying contracts, administering them, automating workflows so that your protocol can keep operating smoothly, they're all essential, but they all introduce um gaps and places where human human error can can occur. We're sort of assuming and counting on the person, the operator, to know what is to know what is right in what can often be very complex series of transactions, and it doesn't really have to be that way. We should put in the guardrails so that we ensure that whatever we are sending on chain, uh we have the

confidence and the proper documentation uh to make sure uh that you're going to stay stay safe. To prove this point, I don't know. I I know anecdotally, but I tried to find some some numbers to support that. We're seeing six to seven times more people or process type exploits, whether it's an exploit or not, it's debatable, uh than traditional code bugs uh or or traditional code exploits. And I think that shows something. I think it shows that we as an industry have gotten a lot better at protecting our code and making sure that the little house that we built is locked down and secure,

but we haven't accounted for the complexity that comes with having many of these protocols interact together on chain. Which brings us to the second risk, uh the dependency risk. Now, this is um this is something that can happen and can can trigger uh very very very uh severe consequences, and you don't even need to do anything wrong. So, when you when you build a protocol, in most cases, you go and you deploy together with the dependencies that you lean on, and those dependencies have their own, and so on and so on. The web gets pretty large. And at the end of the day, this is one of the core benefits of blockchain, right? This is how uh this

is how we how we got the the DeFi legos, right? But the flip side of that is now we need to account for all of these working as intended within the parameters that aren't going to mess up our that aren't going to mess up our protocol in unforeseen ways. With crypto being interconnected, we need to make sure that we take care of and we account for failures of second order, third order, fourth order dependencies. Um and we need to make sure that we understand what is the impact if things go wrong on these on these layers. We need to know what's the blast radius if a protocol that my protocol depends on uh suffers a catastrophic loss.

The flip side of that, it is DeFi after all, the other aspect to having this web of protocols connected together on Ethereum uh is the fact that these are all financial products at the end of the day. And the market and how it behaves um is an essential piece that can uh make or break your protocol operations. I think about it I think about DeFi protocols sort of 50/50 a split between the code and the mechanics and the algorithms that they designed. And then the other half being the state and the ledgers and the pools that make up make it up and connect into

the rest of the rest of DeFi. Now, this is something This is something that is also a pretty normal, pretty standard uh problem that traditional finance is also trying to solve. But I think they have it a bit easier uh compared to compared to on our side. Uh we don't really have the luxury of having trading hours, circuit breakers, having human checkpoints in the loop making sure that someone can step in and say, "Hey, let's let's give it a rest uh for a little bit before before we continue uh until volatility settles a little bit."

What took days or weeks uh can unwind in an instant. And uh trying to predict and model these types of uh uh these types of outcomes becomes essential. Because at the end of the day, by the time by the time a human recognizes or notices this type of vola- volatility happen robots have seen that way before. And the avalanche is already triggered and it's often too late to to react there. So, modeling and measuring how comfortable we are with these types of exposures is uh the the of the game of traditional finance.

And then lastly, code risk remains as important as ever. This is the one that we have spent as an industry most time uh thinking about, trying to solve. So, audits, comprehensive testing, managing the surface area, these are all very important and still remain to remain very important disciplines. And we should really keep them up at the same level as we as we did so far. Why? Because well, we're we're not only protecting our own protocols, right? If I'm if I'm pushing something on chain, I want myself protected. But we're also by introducing bugs, exploits, um and increasing the the the overall

level of risk with our protocol, we're also impacting others around us, right? The sort of the entire point of the the dependency risk. So, the main difference here is what I would like to to see more of is instead of focusing on comprehensively testing the protocols themselves in isolation in our small little sandboxes, uh we need to start thinking a lot more about um verifying and testing out on-chain dynamics too.

Not doing this was one of the one of the reasons why our example from the top occurred. So, correct at launch doesn't necessarily mean uh correct at scale. We cannot theoretically model all of the possible states in which our protocols can be in. So, accounting for the world around us becomes important. So, with these four, process, dependency, market, and code, if we step back, we will notice that traditional finance has been dealing with these types of these types of risks for decades.

Um and they have well-known uh frameworks that give them, you know, extensive instructions on how they should manage and behave when exposed to these types of risks. In crypto, we just name them. We haven't solved them quite yet. There is one missing piece that ties all the four risks together. And this is my pitch, what we've been doing for the past couple of years. Um all of the examples from above, all of the types of problems we're encountering on chain, they all stem from struggling to understand understand how our actions uh reflect on a live blockchain.

This has been This has been um the problem for the entirety. Um how do we simulate the exact outcomes against a live state or without actually exposing ourselves uh to the blockchain? You know, one of the one of the downsides of blockchain is that there's no undo buttons. So, before we actually commit our actions on chain, we need to do everything in our power to make sure that uh we're doing so with full knowledge of of what are the theoretical limits and what's going to happen.

At the end of the day, all of the all of the serious industries already approach uh that that have safety in mind, already approach um their their industries respectively. Um an airline is not going to put uh a pilot in the sky without hundreds of hours uh in a in a simulator. Right, financial institutions are required to run extensive stress testing for each new um product that they put out into the market. And it's interesting for for us with blockchain is that we [snorts] might have a unique advantage uh all things all things considered.

Because blockchain is ruled by code, it is designed to be sort of the the world computer. Um what others have been have been doing by trying to replicate or model or approximate uh and call simulation, um we can go to the real to the real thing and try and get exact um exact uh behavior and exact um type of preview uh before before going on chain. So, it's not all it's not all bad things um that blockchain gives us operationally.

So, we have been calling this internally blockchain operations as a broader category. This is something that you know, people the the the people at at the frontier of the industry have been doing in one one way or another for for a while now. And right now it remains sort of a quiet advantage that they can benefit from today. But if we want to get serious, if we want to take on the volume that the institutional interest is going to give us, uh we need to take this seriously and we need to count it as uh as table stakes.

So, going back to the beginning of the of the talk, we've done a great job of building the rails, innovating, of placing all the building blocks in the right places. But we need to focus for the next decade on making sure that uh we know how to operate it at the level that the the global type of attention requires. Thank you. >> [applause]