AI-Powered Auditing: Hype, Reality, and the Future of Web3 Security

Transcript

[applause] So, let's go. And next up, we have Jack. Jack from Sherlock. You all are familiar with Sherlock. And uh we have Nico from Expo. Uh Expo is u like the number one hacker on like the the the leaderboard of some of the web two competition. So, it's awesome to like have Nico here. Okay. And u Okay, let's get started. Uh maybe you all can do like a quick intro on like what is your tool and like um what are

you trying to solve? Like maybe start with you. >> Great. Hi, my name is Connie. Um the tool suite that we build out is called the Olympics tool suite. It's five different tools currently, but essentially the point is to find as many findings inhouse um via a layered security approach ahead of going to an external audit. >> Yeah, go for it. >> Hi, uh my name is Kiel. uh our tool uh essentially I think that uh most of the tools uh are focused on the pre- audit stage uh such as ours and uh uh we also try to find as many issues as possible before the audit and essentially the idea that we really believe in it that

if we find let's say 70% of the issues before the audit your auditor will have three times more time to actually find something much more serious that they who previously. Thanks. >> Hey everyone, I'm Jack. I'm the CEO and co-founder of Sherlock. Uh Sherlock is really focused on life cycle security when it comes to securing blockchain code. And so we're building an AI auditor to help with pre- auditing, basically saving you money so you don't have to do multiple rounds of audits as well as checking your code once you're live on chain to see if there are any new vulnerabilities that the AI can find to save you maybe millions of dollars in bug bounties.

>> All right, so uh my product is called Expo and we are basically focused on web two. So I have no idea what we're going to talk about in terms of web3 but uh we basically do uh AI autonomous pentester. So basically the product can go and find vulnerabilities both from a blackbox perspective basically looking at your website without any information or you can provide it source code and and any sort of like development data and it will take that information to do a much more educated penetration test. >> Awesome. So uh so we have seen like the AI models getting really good at what they do right and uh there there are two trends AI models that use tools and find bugs and then the then there are

entirely pure reasoning models that just look at your code base and find bugs. uh maybe you can share thoughts on like what what do you think is going to win because recently uh I I was listening to this podcast from Dave Del who leads like security at OpenAI and their product adog uh relies mostly on the reasoning side of things and uh as a traditional security guy I do reason but I also use tools right so are we looking at a whole different paradigm of finding bugs using AI or is it a hybrid mix?

What are you guys finding while you build these tools? What approaches work for you today in web 2 and web 3? Maybe I can start with you Nico. >> Yeah, so uh we basically use a hybrid approach. Uh so basically we let the LLM go and attack like a unit of the application. So we try to basically one of the big problems with with LLMs is that uh they're very undeterministic, right? and you're expecting that every test you do has to be deterministic. So we separate like the the attack surface into units. In this case for us is like endpoints and then we send agents then go and perform that attack. Now uh the agents basically have all the tools that a pentester will have. So like you can run like SQL map or any sort of like open source tool that are out there can

use that but it can also generate Python script and came out with their own like sample test to do it. So we give them the freedom to do the two things tools that obviously we have to build some tools to specifically work with LLMs. There's a couple of tools that we have to build in house for example like a headless browser that we use to be to navigate a website and do performs attacks. But in general we give them like the two things like go and per write your own uh sample scripts to perform like attacks or just use some of the tools that are out there the the tool that pentest you use to perform the attacks.

>> Okay. Awesome Jack. Yeah. So right now we use a lot of reasoning um but we do use tools too. So we've built you know static analyzers into our kind of reasoning and I expect that the hybrid approach of using reasoning and using tools is going to be even more and more the approach that teams are taking. Uh, I think doing invariant testing, doing fuzzing, doing formal verification are really obvious ways that an AI is going to be um, you know, working with tools to find bugs in a codebase. So, I expect that to quickly be something that we see some of the reasoning approaches get more and more into.

>> Yeah, go for it. >> Yeah, I would say uh like pretty much everybody is using hybrid approach. uh uh but uh with uh reasoning I would say it's just uh we can get there faster like to higher level of coverage uh because like ultimately I view the LLMs as a uh processor for the high level tasks that we give but anyways we uh break down the entire workflow to multiple stages we use LLMs to resolve essentially solve each part of the task separately and the LLMs which are smarter essentially they just do it

faster more precisely and in the end we can get faster to this higher coverage and low positive rate >> right awesome >> um okay I don't think it's a one-sizefits-all meaning I don't think the right question is what's going to win uh and the reason I say this is because when I look at the historical exploits just in the last 365 days or in 60 days alone. I can see that um maybe one of them would have prevented with our mutation testing tool, one of them would have been prevented with our static analyzer, one with our agentic orchestration system.

And so I believe the correct answer is a layered security approach is the only approach which inevitably leads to okay I agree hybrid. Right. >> Right. Okay. Hybrid. Okay, cool. But um what what do I mean when I when I say hybrid and how do we make that really easy to digest? what you see is what you know when the data set is only historical exploits when the static analyzer behind there is slither and it's you know it's just it can't go deep it doesn't have the right IR and infrastructure um >> when when you don't have all the information present you can't reason aggressively I think would be the answer so if you want to reason aggressively and find really >> deep findings I think you need more

information and I believe that that information comes from advanced tools that take a long time to build. >> Um, and I think historically those have been very hard to gain adoption. People don't want to implement them. But if you can make it easy for an AI agent and something exhaustive, something smart, something quick to take it and and reason and find if the vulnerability can be proven, you're looking at a whole different structure of finding vulnerabilities ahead of the audit. You're almost able to overlap at like 60 70%. Okay? which is a paradigm shift completely, >> right? Yeah. So, uh I I'll play the devil's advocate. Assume I'm like a frontier lab. I have so much money. I have many GPUs. There are machine learning techniques like selfplay, RL, uh reward. You can do a lot of training

with very little data with synthetic data. Given uh like a specific kind of a bug, you can generate multiple variations of it. you can teach your model with RL uh what kind of a vulnerability that is and get really good with the selfplay RL kind of a paradigm and that is how they got good at coding right like if you use LLMs for coding today they got really good really fast because of this these kind of advances in machine learning so I would argue that maybe we really do lack a lot of data like the web two has today. Uh because web two

security has been around for a long time. But these advances in like machine learning can maybe make these uh based models that we use uh also really useful for like web 3 security. Do you do you have a any thoughts around that? Well, I guess if you're saying maybe the base models will be the ones who are best at finding vulnerabilities >> and not some of these tools that people on stage are building. There's a lot of missing data for those base models. I think like Sherlock for example, we have thousands of vulnerabilities that those base models will never have access to potentially.

And so you can still get to a high level of effectiveness with the base models, but you might say if you can build on top of them and add these other vulnerabilities, then you can always maybe be a step ahead or at least be a step ahead early on. >> Makes sense. Yeah. So it's the niche that you guys are like in that makes you better than like the big labs. Is is that a general consensus? >> I I guess I'll I'll piggyback on on Jack, which is he has unique data, right? And that's the claim. and and I say I have unique information because my tools can abstract unique findings about the codebase and then from there we can >> okay great but I I think the the overarching point is we're living in a space that's different than web two and

what does that mean it means you in here you can't use probabilistic security approaches because we can't afford in this ecosystem to be 1% wrong because the incentive structures when we're 1% wrong North Korea comes in and takes that 1% right >> and they have 100 million of seconds right right in in in web two it's >> there's there's different instruct in incentive structures and and security is the the buyer comes and the CISO comes from regulatory posture pushing that down. Um and I'm sure you'd agree with me like yes of course data is at stake and and there's a lot of stake but it's not the same as your company going under within a hund00 million exploit in the matter of seconds. So um a layered approach here yeah is is is important and anyone who has unique data gets us closer and closer to that percentage and

the way you can measure that internally is like how many tools you use and how much effectiveness does each tool have. >> Right. Right. Right. Maybe Nico, you do you have thoughts about that? I think uh there are some comments here made that web two is more like CISO. I I will definitely talk through like our experience with like foundation models versus training models and this is web two so like might not relate with web 3 but uh what we find out is like when we we did a lot of experiment on training and we end up figuring out that like the next few models is basically beating anything that you're building like and this is it goes with like sonet uh jgbtdropics six months ago uh grog was like the best model to find bugs and there was like we

were very surprised about that fact now GBD5 is like the the very best and like there's a big big jump there. So we have been one of the things that we have been doing and is generally you know recommend across any AI company is like having like a very strong benchmarks that you can actually go and and test against it and we have seen that progress between like things that we have built oursel open source and like the models as they're coming and it's really impressive like what is coming out of like each of the different like new models. go. So I'm still betting on the foundation models. I do believe still that for certain type of bundy classes and that would be the case of web three and models might not be as trained as well as in web two than web three for sure is that for different

type of like classes I think that like training models there's like there's an interesting result there especially around like the kind of like business logic vulnerabilities that are not very well defined and is more related with like the context of the application uh and how it's being used for those I think that a combination of like foundation models and model that had been strained for specific type of classes could be like the solution for some of them. >> Okay, >> that's what we have seen at least. >> Awesome. Uh also one of the things uh you brought up like logic bugs, right?

How hard is it for an LLM or an AI agent to like find those deep logical bugs that is easy for a human auditor here to like find these bugs? Cuz I feel like that kind of intelligent intelligence is different from what an LLM you get from a base model, right? >> Can I jump in? >> Yeah, go for it. >> Yeah. The logical box is uh exactly those types of bugs that LMS handle much better than humans because they see the discrepancy between the description the uh documentation and the actual code much easier. So this was the first uh case that we found out when uh top tier auditor has missed an issue that has been found by super immature version of

our software like year and a half ago. >> This was just simple logical. With good documentation, LLM could find the difference between what was documented and what's in the code. Okay, awesome. >> I would say like LM are good at finding logic bugs. The the hard part is like validate them >> like like figuring out what is what is what is the like separating the needle from the hstack. What is the true positive? What is a false positive? That's where the real challenge happen with like I doors business logic authorization issues and like if the end result is not a shell but rather like an information leak you need to like the LM need to be able to reason enough to understand that this is an information leak and it's really complicated because like an info leak could be different things for different type of companies

and even for different type of software. So um what we find out is that um the especially GBT5 the high reasoning models are getting like really really well at being able to like validate business logic and being able to like is not and um you know we we review the bugs that we are finding them with our pentesting team and sometimes like even for us it's hard to say like is it really a vulnerability or not like the in the end like if you don't have the documentation it's kind of like a educated guess and is doing similar things

>> exactly but you had some blog post out there uh talking about some of the new bugs net new bugs that the LLM has found right so what was the validation process was like cuz I'm Sure, everyone here is familiar with LLM spam like you know is this is a bug this is a bug that is a bug and how do you even tell like what is a bug and I feel like Jack runs like like Sherlock which is an auditing platform which receives a lot of fun LLM bugs how do you go about dealing with that >> yeah so the validation part I agree it's extremely difficult and I think Sherlock is the only one up here that has probably validated hundreds hundreds of

thousands of bugs through the audit contests and the bug bounty platform that we run and it's extremely expensive. You need some of the top human experts in the world and you need to pay them thousands of dollars in order to properly validate these bugs, especially these logic bugs. Yeah. >> Um to figure out if they're real or not, if they are in line with the spec that the team provided or not. Uh and so this is a huge cost to finding logic bugs with AI auditors is you actually need humans for a very long time to validate those bugs and it's expensive and takes a long time.

So uh I fully agree with uh what you you mean and um but at the same time there is an interesting point uh uh is coming when so currently for instance uh for validation we encountered it recently that uh cursor with good context is really good for validation specifically and like this is a kind of funny case when I blindly triaged like all the findings via cursor submitted them to another competition platform and I got seven out of eight accepted. Uh I didn't look at them at all but but we are really close to the point when uh AI is

saying something like this is correct because of this and that. Human is looking into that and saying okay in some cases yes in some cases no and then we do another approach when we look something something is not what we expected like annotation is not matching to what we see like the benchmark starts uh uh kind of uh saturating and we get back look precisely and see oh wow this is we who made mistake not AI and And this is a point when we looking into the data back in what we used for our like let's say not training but at least

benchmarking and we notice that there are some new findings that have have occured while we've been predicting. We mark them as false positives and try to re get rid of them and they are correct [laughter] and that's insane. >> Yeah. What we generally do for validations is uh as as much as possible try to take it away from the LLM. So for example uh for for I don't know again about web three but for web two like if you find a crosses scripting SQL injection etc you you can have you can build something that basically validate that vulnerability just to give you an example like when when expo finding a cross scripting basically we have a validator that's headless browser that will go and navigate that website and if

we see something like a popup coming up say like yeah fantastic these are crosses scripting and I mean and you constantly are improving those validators because the better the agents the the LMS are the better at cheating those validators that you build in place but so far we have been like very um um very good at validating most of the bulbs. Now the business logic the the other type of vulnerabilities are the ones that are complex but we're we're so far seeing like really good results for for the new like high reason models being able to like uh reason about those validation and getting us like a decent amount of like uh true positive false positive rate.

>> Awesome. Uh Channy, did do you have any comments about >> I actually don't I feel like everyone really covered it well. >> Okay. Very cool. Very cool. Okay. So, uh we covered a lot about the capabilities of LLMs and I feel we are all bullish on these things getting better and helping us do our jobs. So, recently there was a blog post from Anthropic. Uh they their models were used in a nation state espionage. Uh so it it looks like uh these offensive cap capabilities of these LLMs are are are used today and uh the attackers have access to these today. So as the builders of defensive

tools, right? uh how are you helping at what stage of the SDLC process does your tool come in and like how are you helping the defenders and what do you think uh are the attackers getting better at this using AI for offensive attacks cuz if you ask Chad GPT to build an exploit it would just politely decline so if you but you can ask it in creative ways right so I think that is what is going on but you guys are the experts. So what what are your thoughts around that?

>> Yeah. So I think it's an asymmetry problem. Um specifically about our tools, we s the SL, you know, SLDC life cycle. We sit before an audit. We believe internally teams should decentralize security and again this is for web 3 ecosystem exclusively and that they should try and find as much internally and then you have the benefit of uh great audit firms that not only have audits but also run their tool for the teams um ahead of competition. So, so that again adds the to the layered approach. But when I talk about asymmetry, what I mean is when a team is building a protocol, their job is to market the protocol, to raise money, to hire people, to do a million things and security is this much part of it, even though the consequences are >> Yeah.

>> are fatal essentially. Okay. Um, now when you think about the person sitting in North Korea, their only job is hack this protocol. And again, going back to asymmetry, their reward is so much greater, >> right? So if they just get one exploit, they can pay for, you know, their year plus their family's year plus, you know, and everyone loves them. Um they they've done great. Uh for a founder, the consequences are fatal, right? But not focusing on it, the incentive structures are like also they have to hire, they have to raise, they have to do so many things that it's hard. So it's going to take a little bit more time for the industry to even out and people to begin to really implement this more aggressive layered approach. But I really have seen a shift in the last three years where teams are realizing

that you know relying on just one thing won't won't lead them to success and and that's just as important to them as you know hitting their product metrics hitting their sales metrics etc. And as we see that shift in industry >> hopefully over time um we'll be able to line in terms of like our technology catching up and um it being implemented and so robust that our our protection systems will protect us from >> Got it. >> the person who's just focused on that. Yeah. So, do you guys think it's about who has access to like some of the best models or is it a skill issue?

>> Go for it. Do you guys >> Okay. So, what I was going to say first is that um that that report from Antropic is kind of like the tip of the iceberg, right? Because like can you imagine nation state using like Antropic or OpenAI for like doing the test? >> That's not real. >> Yeah. I I know they know if someone is listening to you and entropic or open AI. So like nation states attacker like Chinese nation attacker attacking your your your environment from that point of view to me is like could happen but it's not a real team like the real offensive team will be using their own models and we know that China and Russia are building like good models as good as what we are seeing in the US. Yeah.

>> So there's things are happening and they're happening fast that's for sure. This is the very first example of this. Uh but again to me it's like a weak example because like we know there are much worse capabilities or better capabilities depending how you look at it um on the nation state side right >> um having said that I think that like people like like us that working on like AI and and and and offensive or in security in general we do have a responsibility to be able to provide companies with tools that match that what others are building on the like dark side of the internet. I I think that there's nothing different from what we're doing for like a human can do. I think the big difference is like speed

and scale and I think companies needs to be aware that uh like now attacks are coming at a scale they had never seen before and at a speed that had never seen before. Like you used to connect a a computer on the internet and like 10 minutes later they will have some some tool trying to like you know test for like default username. Now someone can do like a full attack on your on your environment completely automated, right? And that scale is something that we have never seen before. >> Yeah, exactly. >> Yeah, I definitely agree that you know to what Kani was saying, there's an attacker's advantage where you can defend 9,999 attacks, but if they get one, all they need is one and they win.

>> Um, so they have that advantage. I think we as builders and the good guys have a lot of advantages as well where, you know, we can go raise millions of dollars from venture capital and really put a lot of effort behind this and sell it to a bunch of different clients that will pay us money for it. And so that gives a ton of advantages and scale advantages in terms of building an AI that maybe Russia or North Korea doesn't have the resources to to build. And so I think that gives us a lot of advantages as well. >> Yeah. Great. Yeah, I would add to this. Uh in long term, longer term we uh have advantages uh over the hackers. Uh but at the same

time on shorter term they they actually have have this uh capabilities already here and uh they can spend some time specifically focused on this uh like potential uh this particular explorator that they they can uh scale them already. they can automate a lot of stuff. So essentially now is the most kind of dangerous time for the project that have been already shipped or that are shipping steel in the same manner in the same pro processes related to security. So in longer term we as builders uh we have uh a lot of means to prevent future attacks. So we we manage

our risks and we we can use uh the mech some specific mechanisms in code in operations uh in uh financial incentives and so on and so on to make this extremely hard and extremely expensive for hackers to like using whatever tools to exploit them. But at the same time right now probably we in a way we we are losing because of like for instance the balancer hack I I really like five years the code was never touched. It was audited 11 times and uh all the essentially we we read that it was out of scope here and there but like come on

it it like it was audited multiple times. developers are top tier developers. It feels like it's just out of our capabilities to have like code that has no no vulnerabilities at all and now hackers got access to such a great technologies to find those those smaller issues. So in the end like I believe that this particular exploit was related to AI and we'll have we'll see multiple uh like huge exploits in near future but in longer term the industry itself feels like more of a kind of antifragile and uh we'll survive and we'll develop.

>> Got it. I was I want to say and and just to bring a little bit of controversy here is that I think that attackers need to be right more than once and I think that if you do the right things that should be like the like scenario because like they have to be right one time when they get into your network but then they have to be right every every action they perform they have to be right because otherwise they will get detected right so like once they yes once they able to like find that bug that compromise you that's the one time in like 99 that defenders are trying to defense but then once they're in there should be layers of defense that will be able to like have them like needed to be right in many many different ways in order not to

be detected and that's sometimes that's what we have to aim for. >> Yeah. So the recent attack Krill was talking about it it was code that was live for five years and then we all thought it was solid and then we turned out to be not true. Uh it's similar to like how in Linux kernel the code lived for like 10 years and then like someone found a bug today right so uh Jack you had to you wanted to share something. >> Uh yeah I think there might be a slight difference there between web two and web 3 where web two you have more ability to put filters in place hopefully multiple filters where in web 3 >> it's one transaction you know one transaction and you are 20 billion gone.

So it's it's a little slightly different I would say. >> Yeah. And you have that luxury of like the CISO in place and the CISO is your buyer and the CISO has a full stack of tools that that can assist your tool um and sit in the middle. Whereas for us buying posture is kind of determined by what that protocol feels like doing um and how important security feels to that protocol or how much perceived like status quo or what they need to do they're doing. Um, and it's it's really hard to inform because there aren't regulated standards. So, you're essentially like living off of, okay, here's becoming the new best practice and once enough people jump on that ship, then great, more do. Um, but even like when I started in 2022, the big wave was onchain monitoring and no one

was doing onchain monitoring. Now, pretty much everyone does it and it's like if you don't do it, something's wrong with you. And now the new wave is like tooling ahead of audit, whether it's Agentic, whether it's um, LM wrapper, etc. And now we're trying to figure out like what's the best practice and then eventually I'm knocking on wood but everyone will be doing it. >> Yeah, it's like web three is speed running the speed running the web two playbook. We we we will get there and we will hopefully have layers of defenses. Uh another thought about all the audits we get and we still find bugs right so formal verification comes up uh when we discuss these things and with LLMs I have seen hybrid approaches where like you generate lean spec of programs uh

something like CVL or like lean uh what are your thoughts about uh combining formal verification and LLMs to get a higher assurance for the code you are auditing. Uh maybe if anyone has done anything like that or has thoughts about it, feel free to share. >> Yeah, I think it's 100% on the road map. I'd be surprised if we're not seeing it in the next 6 to9 months in terms of AI auditing tools integrating with formal verification. I think the two big problems with formal verification, one of them is that on really complex code bases, it takes too long like the mathematically it takes too long to run and so it's not feasible. The other

problem is it requires a ton of either developer or auditor effort to come up with all the rules, all the invariants and and and create all of that. And AI can really help with that second one. So I think formal verification and invariant testing and fuzzing kind of get superpowered with AI. Yeah. >> Uh I would add that u uh for with formal verification this there is kind of unique uh opportunity to if uh LLMs are uh very precise in programming. So very accurate. This is essentially what we are getting every every new release. So at some point uh you probably will have

available uh the model that can write. so so much code that covers so many environs that is impossible for any human to do. At the same time uh in formal verification the uh level of uh let's say co cognitive uh resources or intelligence that is needed to run it and in at least in some attempt get fully covered fully correct uh uh program is much higher than just in programming or just in testing or fast testing even fast testing. So I would a bit u uh doubt about uh 6 to 9 months

[laughter] but let's see. >> We'll see. We'll see. >> Um yeah I I'll just tack on to that. I think forget AI machines in general are good at exhaustive tasks. Tasks that a human maybe could do but it is would be you know take you know 3 weeks as opposed to uh 30 hours or 20 hours. Even though it's exhaustive computer intensive it still is a lot quicker. Um and uh the core issues with and then just coming back to like formal verification is like one it it asks you to learn a second language, >> right? >> Two, it assumes you're smart enough to infer the right specs. >> I don't think everyone's the top 1% which is like really what it takes to infer the right spec. Um and I I think statistics would would agree with me. So if you can find a way and like listen if

you can pay for Storra to do that for you, fabulous. And Stor knows your codebase. Majority of us don't have that kind of budget. So if you can find a way to hit those invariants and um infer those specs with something that reasons correctly and infers the right logic and maybe there's some like intermediate piece where you're um putting that in human language or you're um verifying its assumptions and there's like an interactive piece that's fabulous that adds another layer to your um like I think of it as like a mathematical score to your percentage um and confidence in like I have tested everything and ultimately with security it's like what percent have you tested and that's how you can go and deploy to uh go and deploy confidently is like the more things you knew you did and the more the

more boxes you know you checked the lower the chance it is to hack you essentially. Yeah. >> Nico, do you have any thoughts around? >> I don't have any thoughts like in web two like you cannot really do formal the speed of of code like you cannot really do formal verifications. >> Okay. I guess we have five more minutes so I guess I'll uh have one more question for the panel and then maybe we can take some questions from the audience. So, uh, one last question. As you build these tools, what is your number one pain point? And, uh, what what do you think, um, like maybe is it solved by like a next LLM update or like what what would it take to like solve that? Uh, maybe I can start with you.

>> Um, yeah, so again, we take a very different approach than uh the other teams here. So I'd say the hardest thing is just building these these core tools. Like we have our own symbolic execution machine. We spent years building out our own intermediate representation. We don't rely on the slitty compiler. There's there's so much there that is just so expensive and such hard fundamental problems to solve. Like we all spend our weekends reading books about compilers. Like that that's the truth. It's not it's not easy problems to solve. And like building those core tools that sit under the agentic orchestration is like the the hardest part for us. Okay. Um, and just building that fast enough and at a speed that it gets in our team's hands, like not our our personal teams, but our customers hands. Uh, so that's the biggest thing we struggle with. We don't actually

struggle as much with like leveraging AI to make it more digestible and more usable. Um, I think that's a a much easier problem to solve. >> Got it. Krill, >> uh, I would say with the current level of capabilities of LLMs, uh, there's already so many things that we can build. uh but it uh as you said it it takes uh so much time because of the cost of the the biggest models that already have capability to do something and time. So uh you you you run one API call, you put a lot of data, you see how it works, you make your assumptions, uh next hypothesis, next run, run and it costs so so much money

and it spends so much time each iteration. So like we could could achieve much better if uh even with the same models just like working on on it. The other thing is I already mentioned is that uh the annotations of the missing annotations. So the things that we think that ground truth for the um audit reports that we have or competitions they still contain some things that were not uncovered. AI is uncovering and we don't understand. We we cannot automatically find those things.

>> Right. Yeah. Jack. Yeah, I talked to our team about this recently and I think it comes down to useful context. Uh so one example is AIS are getting better and better at having large context windows. So you can add a lot of context into them, >> but it's not a useful context window. Like it's very very bad at being able to usefully um use all of that context even if it even if you can give it to it. Um, and and the other side of useful context is just being able to like there's a a divide between the training and the inference and it's not quite like a human would remember the flows and have the same type of memory as it goes through a codebase and sees these vulnerabilities. So useful context is

how I would describe it and that's just kind of getting the base models better essentially you know. >> Got it. like yeah >> um I say for our case is uh like our own speed like we have too much things to build that we don't have enough time uh obviously if the model gets cheaper that's that's good because like that's going to be that you can >> uh offer the off the service more more broadly and we and the expectation is that compute was going to get or inference is going to get like lower and lower as as we go we have seen it last year from this year it's >> it's insane the the the cost price so we we'll see that tendency and we know that's going to happen but for our in our case is mainly like there's so many things that we have this is kind of like

a new era or a new kind of like market that is being built and we need to like build fast enough to be able to serve that market at least from our point of view. >> Awesome. All right, we have one more minute. So if anyone in the audience has a question please raise okay you wanna you want to speak to this mic. So the uh the web 3 is I would say it's arguably simpler when you compare it with web two because you said you cannot do formal verification one on web two but you can do it on web3 so because the number of operations are much simpler the you stack is simpler you can model memory and so on uh but when you start building these AI tools

uh you have systems which you can test um in a very separate area. However, because everything is composable in blockchain, you do you also think about modeling other systems like uh unis swap balancer and other systems that the code you're um auditing right now has to can interact with in the future. Um yeah, I think if you're trying to audit a codebase and you don't have the connection to those integrations, like let's say you integrate with unis swap or you integrate with a you're going to miss vulnerabilities and you're going to miss really important things. So that's one of the big problems I would say is that having that big big context window

is super important and finding any way to you know localize where you're looking for vulnerabilities is really important but so far I haven't seen any good approaches to it. You really just want to have everything in context. So if you integrate with a you want to have that a codebase in context or at least the chunk that you interact with. >> Awesome. All right. That's all the time we have folks. Thank you so much for joining me here. [applause] Hello.

>> Hello everybody. So in the next panel, we're going to talk about some of the crypto startups and the facts that they lack security leadership. And then we're gonna explore uh the risks of it and the benefits there are to build an official a formal security program from the beginning before the bridge get to you guys.

So I'll invite the panelists. Um is there an order? Yeah. Ryan, Ryan from from Gauntlet, Joe >> from Mandant, Chris >> from Polygon, and Edido from Blockade. >> Hello. Hello. >> Halfway through we do musical chairs, right? We just swap.

All right. So, uh, let's do a quick round of introductions. Uh, do you want to get started, Ryan? >> Yeah, sure. So, I am Ryan. I'm the head of security at Gauntlet. >> Go ahead, Joe. I mean, I going really. >> That was a very elaborate introduction. >> You can go further if you want, but >> tell us about Gauntlet. >> Hi. Hi. My name is is Joe Dobson, also go by Dobs, and I do threat intelligence at Mandant. And what what that means is I identify and track bad guys as they move on and off chain. Right. So also looking a lot at off-chain infrastructure.

>> Cool. So I'm Christopher. I look after Polygon security, specifically Polygon products security and as well as the smart contract engineering team. >> I'm Ido, one of the co-founders at Blockade. Blockade's a real time onchain security platform. We work with some of the biggest exchanges, wallets, protocols, asset managers to protect various different um uh things they build and run on chain. >> And I am Thiago. I'm a technical director at the crypto isac. Basically what I do is I try to find data on and offchain try to see how that data can help uh protect and prevent events and stop events uh incidents uh as soon as possible.

All right. So uh let's start with the topic right. I mean we know that a lot of the uh security uh the crypto startups they claim to to have security they claim to be a security first uh institution or organization but is that really the case so I'd like to start like with a you know provoking the the the folks here to talk about like what's the real uh the you know what's the real security scenario for these startups and uh what's you know what's the reality and what's the, you know, the the the philosophy that they they they sell around.

>> Yeah. You know, I can jump in there quick. So, I've been in crypto for about five years now, and I've gotten to the habit where every time I see a hack, one of the first things I do is I look up who's the head of security at that company. And if I don't find a head of security, the next thing I do is, okay, well, who's the security team at that company? I'll actually go and reach out to my network and try and figure this out. And it boggles my mind how often there is nobody or it's like an engineer who is double duty. Like here's my engineer who is also the security guy. I'm like if you're the engineer, you're busy. Like there's no way you are also doing all of the security stuff. Um so I've really been probing a lot and that

that was kind of like why we wanted to talk about this is how many companies out there are actually going and building security teams. Um so go ahead drop. I I I think that part of it is in my opinion and you all can tell me if I'm wrong. >> I will. >> Security is not the default, right? You have to be intentional about security. It's not going to happen on its own. And so if you do not make an intentional explicit effort to have a security practice to do SEC, you know, things in a secure way to to understand OBSAC and best practices, then you're not going to be secure. you are a a target waiting to happen, right? A great example of this is if if you fund a new EVM wallet, how

long until you get your first fishing token? Does anyone know how long it takes? Less than eight minutes until you get your first fishing token. Now, imagine a new user, new to crypto, brand new wallet, first time they ever used crypto. They fund it. within 8 minutes they're already getting tokens sent to them that will drain whatever money they put into that wallet if they interact with them. We have to keep that in mind because the the default is crypto is kind of in a bad neighborhood, right? You need to have a security. You need to have best practices.

>> Yeah. Know I I completely agree with what you both say over here. No, it's it's also it's a difficult cell. No, that's that's a problem really over here is like if for anybody that has worked in in security similar to people that have worked in in research for example, it's difficult to tell like let's invest in security you know especially if nothing happens if everything's okay if you know the end of the year comes and you know my my boss or the CEO of the company or the founder is like hey Christopher you know like um yeah I don't I don't think we need this security thing like what what what what is the return on investment? What do I get about it? nothing happens and that's that's a very difficult cell because how do you sell the fact that nothing happens is the benefit know a lot of

people don't get it they want they want that you know energy that want that you know scrolling ticking you know thing that says like oh amazing great job good job but yeah and security is not really like that >> so so Christopher though talk to me about Polygon security team because I think Polygon has one of the stronger security teams in the space I'm actually kind of jealous like I look at your team I'm like I can't build that at go talk to me about how did you convince them to build out the team. >> I don't know. You should tell me. You were there before me. [laughter] >> Yeah. >> Is it so? But still tell tell me about the uh >> No, I I think I think you start seeing the return on investment when uh you know, you actually see the amount of

like like crap that go through your network, the amount of bug bounties that you start paying, you know, when you start to see really the consequences and you see the consequences over time because they start accumulating. Okay, one fishing email, two fishing emails, like who cares? You know, Google will will cover it. But once you start build getting employees and you know, you start getting public attraction and everything else, that's when things start to accumulate. No. And and compound the amount of fishing emails, the amount of bug bounties that you're paying, the amount of bugs that just the engineering team is, you know, having to fix that basically slows down. So then they start realizing like hey if my software development life cycle is starting to slow down because of the amount of bugs we need to fix from you

know what is already out there then uh you know you kind of say like okay maybe you know if we had a security team that could take care of this that could make sure that you know my engineers are focusing on what they need to do you know which is not ideally triaging bugs and stuff like that then it's when you start to see that return of investment and the same for fishing emails and stuff like that you don't want your IT team to be looking and triaging fishing emails realizing, hey, is this legit? Is this not legit? You know, you want a dedicated team to do that so that it can continue helping the business move forward.

>> I I think though one way that security can provide a return on investment, right? It's still going to be cost center, right? Even web 2 security is a cost center. We we have to accept that. But we're in an industry that runs a lot on hype and I think that security has not taken advantage of that. were a victim of the hype instead of being, you know, instead of being a victor of it in that if we had better sharing, if folks would come out and say, "Hey, we were targeted by XYZ and here's how we stopped it and demonstrate thought leadership and security that would build some hype. That would build some return on investment because that's what leadership wants. They they want that

that attention market share and that would help. I think >> I I want to ask one thing. You you mentioned you know security is like a cost center, right? It's been a cost center in web two. It's been a cost center in web 3. You know, I I think one thing that's really interesting about our industry specifically is that we're all building at least today, we're all building ultimately financial infrastructure like all of us, right? And at least as a user when you think about like why use one piece of financial infrastructure versus another piece of financial infrastructure. I don't know what your answer would be, but mine is generally it's like I don't know. It's a thing I kind of trust, right? It's like this thing holds my money like it was there yesterday, it's going to be there today. like that that's why I go and use that thing,

right? I think like we're in Argentina, right? We were just speaking about like the the quos, right? Like before, right? That's I'm sorry for the mispronunciation, [laughter] but but it's like, you know, the the and maybe like maybe someone from Argentina will come and pick me up for saying this, but like you know, you see the local the local economy, right? very much uh you know not trusting other things that we would kind of in the states or in Canada and and and trusting you know things that are more um you know on the on the riskier side or things that we would otherwise not necessarily trust in the United States and so and so I'm curious like your thoughts is like uh like just my point here is you know we're building financial infrastructure trust is a differentiator right security builds trust why is it a cost center

>> yeah so and just kind of like a kind of riffing off of that a little bit. Um, why do we trust our money with protocols that clearly don't have like they have this advertising like oh we're security for security and then like like literally if they don't have a security team like why are we giving them all this money to hold on to and just like taking that massive risk when we're like to your point like we're complaining about other institutions not being trustworthy with our money. But like there should be a way we can check this. Like if if I am going to go and put my money in DeFi into some product or some vault, I should be able to ask them, hey, tell me about your security. And it should be an like there should be some standard that's honest that says, "Yeah,

no, this is what we really do. It's not just a bunch of BS on the web page, right? >> Beyond a PDF probably like beyond an audit that goes and says, "Okay, we're we're good." Right? You should you should have a team. Like you should I I think that's actually a really good like the thing you mentioned in the beginning, right? Like that check that you go and do. It's like who is the team, >> right? >> People should probably go and check that before they put their money in in different vaults and different for different curators for for things like that right? >> I I think people basically assume that no, if if you look at the web two u you know area actually all the all these companies when you work when you do B2B you have these standards you you know you have these frameworks in order to do

security. If you don't do these people are not going to buy from you and by people I mean businesses are not going to work with you. They're not going to make transactions and so on. But on the B2C, in the web 2, you as a consumer, you're just assuming your bank does does security. It's like basic. No, but in crypto, I I personally believe that a lot of people think that protocols do security. That they're taking care of their infrastructure. They're taking care of their employees, of their laptops. But the truth is that probably 80% of all the protocols, if not more, don't have at all a security team as we were talking about it. But people just think it's implicit and it's there.

>> So what do you say to like these wonder teams? And sorry to take these things are just coming to my head here. Like these wonder teams of like their excuse is I've got 10 employees and we've got $200 million in TVL. We're just too small to have a security team. Like how do you how do you make the case for that? Or what do you say to those people? Because I I've struggled when I've had to try and tell them, "No, I think you really need to hire someone fast if you have that much in TBL." But >> I I think it's not about them. It's about the customer and that's who they have to think about, right? Why do they need security? so that they can provide that value, they can protect the value for the consumer, for their customers.

Um, and I think a solution is if they don't want to hire a full-time security person, they they can start out with something like like a viso, a virtual SISO, right? They they can start out with someone that offers, you know, security as a service. They can get some level at ADI of advice and input because I think that it's often that they don't have awareness of just how bad it is. So yeah, >> one one thing a lot of the time people would just outsource for like consultancy uh firms like audits. That's obviously great. I mean I I was a consultant for more than 10 years and every time I went to like some of the big taxs to review source code I wanted to you know deliver my best and really

find bugs and really get them fixed. But uh especially for the the new companies who rely only on third party providers what's the benefit of that and what's the limitation of that when it comes to actually getting and growing an internal team >> right so for first of all for auditing I mean at gauntlet we definitely outsource auditing to the expertise like it's hard to to build an audit team inside a company because auditing is so incredibly specialized at least for a for a smaller place I know some places do you actually build internally? So you have like your solidity coders but having a full-time audit is tough. So we we would outsource that, right? And that's a probably one of the areas where I'm really happy to outsource. Like it

works well, right? Um other things in security I I prefer not to outsource, but I'll let you guys talk about that stuff first before we talk about what's not good outsource. >> No, I I think on on the on the previous thing you guys mentioned around, you know, I'm a team with with 2 million in TVL, right? I think like we as security practitioners like need to understand that security is an afterthought, right? It is something that you bring on after you see some level of like success in your business in the first place. Right? If you have nothing to secure, you shouldn't spend on security obviously right now. $2 million is probably something that you should you should spend on like there is there is uh it's a meaningful amount of money where you should probably be spending on security if if if you have that right and so and

I 100% agree with your with your like V VCO analogy where it's like you don't need to hire you don't need to build these teams. um you probably would be better off like like you know if you know the the amount of money you would want to spend on a on a CISO for bringing on right for for for $2 million better spent on a on a VC so that fractionalizes and does a lot of these different things like for sure I think that uh you know there's you you mentioned like doing that for for smart contract audits I I and I totally agree with you that's that's great but when it comes to like building products or even like an L1 L2 uh if you don't really have like uh an expert inhouse or even like new you know fully dedicated to understanding the development road maps

the team cultures you know their their inner pains what are the most critical components what's missing there is it just you know security testing is it complex design if the if you don't really understand the iners of those development teams and integrate very closely with them you won't be able to help. Right. >> Right. Right. >> So Chris, I I'd love to hear your experience in that sense. >> Yeah. I I I think you know probably difficult to sell but I think the most important thing you to do you should do with doesn't matter if you're small, you're big, you're one person, you're 10 people, you're a million dollars, you're five bucks is just understand what is your risk profile. Like literally understand okay what is the worst thing

that can happen and how could that happen? Just just answer those questions. what what is the worst thing that could happen to me and you know how could this happen and once you understand that question then you start making a plan and figuring out okay should I be spending in people to hire security folks should I just be doing an audit like the worst thing that can happen is a smart contract hack okay then just do an audit you know couple of them hopefully then you're okay but if the worst thing that h that can happen is an employee being compromised a laptop being compromised uh my AWS system being compromised no you need to understand what is that risk profile Well, so that you can actually allocate resources appropriately to that and then measure it know hey if this happens do am I going to lose $10 million then

maybe hiring for 200 a year or 300 a year a security guy it's worth the money you know just need to do those economics >> in in web two one recommendation we we have is that enterprises need to have a threat profile right they need to understand what are the threats facing them what attackers would come after them what kind of tools would they use what you know the TTPs tool tactics and procedures what does that look like and we encourage them to make that a constant reevaluation right you don't just make it once and it's good for 5 years you actually need to sit down and re-evaluate as technology changes as the the the world changes as there's new types of threats out there you need to

re-evaluate what does your threat profile look like so that you can protect your organization that's one of the the sad things about web agree to me is in web 3 a compromise can be catastrophic for an organization but security is not taken near as seriously as it should be. In web two compromise is traditionally not as catastrophic for an organization but a lot of them take security much more seriously. I think this is a really interesting point on just like how liquid vulnerabilities or exploits or like security issues in crypto are compared to like like web two and I'm I'm sure you've like heard of this kind of analogy but like you know it's crazy you find a vulnerability like even a zero day in like uh um in web 2

and if you want to go and actually weaponize that to make a million bucks pretty pretty darn difficult to do like uh um you you you know you have to start like you know building malware around that you have to go and deploy in some kind of operation, you got to find data that's worth money to sell and and or or like you know wrap a ransomware around it and then okay and now you can start kind of going and doing these things. That's that's a ton of work beyond just finding the vulnerability. In crypto it's insane like you find a vulnerability you make a million bucks like it's uh it's basically that simple. >> You know what I find just amazing though? story. Uh this there's this weird thing in crypto that that I don't feel like happened as much in web 2 and

maybe it does happen a bit more where like a protocol gets hacked in crypto, lose a bunch of money, and I feel like a month later they're back at it and it's fine. Like everybody just there's such a short memory in crypto that like you know, oh they fixed their problem. Let's just give them more money and give them another tr like go ahead take another spin of the wheel. Um that to me is like it's it's like they just don't care about reputation as much as I wish they would. Like I I know web companies that would have just completely died after a major hack, right? Um except for hotels because I think every couple months I get a thing saying my private data is gone from whatever Marriott or sorry I probably shouldn't say company names.

[laughter] Leave that out. Company X or Yeah. Anyway, sorry I did. >> You get free credit monitoring for life. Okay, just don't worry about >> much free credit monitoring. >> So we're talking about like the financial side of things. you know crypto startups they are typically born as sometimes with like $200 million already locked in. Uh but what are the other specific aspects that makes these problems very very uh special and critical for crypto companies? Um one of the things that comes to my mind is the like whenever it's done whenever a transaction is signed and sent into the network it's not reversible right so that's one thing what do you guys what else do you guys see that's

special to crypto >> I think uh if you compare between web two and web three type of companies in the web two mostly the worst thing that can happen is like you know some PII data getting leaked uh company reputation being affected um Yeah, those type of things, database being leaked with a lot of you know confidential sensitive you know financial data stuff like that it is definitely catastrophic for a company but you know in the end the users are somehow protected you know however in crypto you are losing not only your money your VC's money but you're also losing your users money in most cases know because all these protocols are open to the public anybody can put money into it anybody can put to tokens into

it and a lot of these People obviously do it with very good intent and you obviously have no idea about what what the risk and and guarantees are over here. And you're actually you're directly affecting the user and that is the the the part that hurts me the most when uh protocols get hacked and everything else is that you know it's it's it's mom and dad and your grandma and the lady that cleans your house that you know with with because they cannot access a JP Morgan account or they cannot access buying stocks. They do it through crypto because that's the only access they had and you just lost that a thousand bucks 100 bucks that they've saved that whole year and that's the the you know the unique thing that is also horrible know when there's a hack to to Chris's point I think the long and the

short of it is the only real difference is the crown jewels change for the organization right I think you said this you said this well like it's in web two it's databases it's data like that's mainly it right um obviously the reputational risk of that uh is probably shared across these two things, but the the crown jewels are the data. And in crypto, the crown jewels is money. And so like that's ult so like now now just like okay, what's the difference between having money that can be lost versus having data that can be lost? Like that those are the real differences, but ultimately that's that's the main main change. >> I I think though it's it's going to get it's going to get worse as we uh I don't like to say that >> this is not a fun panel.

>> So sorry. >> We're full of happy people, [laughter] right? >> Um >> so Bitcoin will get even lower. That's what you're saying. But I I'm saying like uh think about as things like deepen take off, right? As we get more real world assets tied to crypto, that means that the implications of a crypto compromise get more significant, right? Unless there is, you know, you know, secure design thought out in the beginning of an expectation when we get compromised, how do we protect these assets? But if if you are storing things onchain or have an integration on chain that has a direct correlation with real world physical assets and infrastructure, what happens when that

gets compromised and it's on an immutable blockchain, right? That's a whole different level of implication that we as a community really need to think about and seriously consider. >> Yeah. I mean, there's some a bit of happy note because it has been a little bit of a a down note. Um, >> sorry. a lot of the the compromises we've seen uh I mean if they just had so I I don't want to tify it but in a lot of cases I look at like if they just had someone there in security you know like it's like it's a lot of really bad messups that lead to a lot of lost money whereas it's it probably could have been avoided almost entirely like I'm not convinced when people come up here and say oh you can't avoid being hacked if

if you're you're going to get hacked assume you're going to get yeah we assume we're going to get hacked because it's kind of part of the game we have to be prepared like we get hacked. But I mean there's plenty of protocols that never get hacked, right? Like and maybe it's because they have a great security team, maybe not. I don't know. In some cases, I know it's because they have a good security team. In other cases, they're just lucky. But uh but like like take an example, the BBID hack. Like that was avoidable. Come on, guy. Like there were there was things that they could have done to avoid that one on several layers. Like just about every single one. Like bouncer they could have avoided on several layers. Like you look like in retrospect it's of course easy for me to say that but someone with a

lot of security expertise probably would have sped that stuff before right but we're also expecting companies to go up against nation states and that's what we have to keep in mind right it is asymmetric >> right and and and so I I think that we need to push folks to do more but I don't think we should shame folks for being compromised ized unless you know there's clear missteps that are you know they didn't do industry best practices they you know they they they didn't care about XYZ like we should push them to do better but we also need to acknowledge it it is a an environment where there's some very strong adversaries

>> I I I disagree so I don't think you should shame individuals but I think you should shame companies because while the individual can make mistake like if you know you're looking for a uh it looks like a great interview and you get pawned by North Korea. Yeah, that sucks. But there are so many things about that story that could have been like did you have EDR on your machine? Like did your security guy set you up for failure? Did he give you training? So yeah, the individual maybe we don't blame but the ecosystem and the companies they should be held accountable like they should >> I do believe there should there should be accountability.

>> Yeah. Thank you. I I'm glad we agree. >> I'm not against accountability, right? I just one one reason why I say all this is because in web two there's not near as much sharing of compromises as there should be and part of that is because if someone shares that they were hacked right what benefit is there for them >> and that's one thing you really have to consider in web 3 there is an incentive to share that there was a compromise first off it's hard to hide it because blockchains are transparent right at least for the most part and so there's some incentive to share because they they want to protect their reputation.

But also, if there's a massive heist where billions of dollars are stolen and that money gets dumped out through OTC vendors near North Korea, that's going to cause a dip in the market. It and as well as it's going to make people unhappy and it's going to make them feel like they they don't trust the systems. And so there's an incentive to share to protect the whole ecosystem. And and that's why again I advocate that if if we make security cool right >> it is cool what are you talking about >> and we make it so that you know when people share how they stopped a hack right or they were hacked but here are all the the details the indicators etc then that can cause a you know a rising

tide lifts all ships right >> raise the floor of security >> yeah so one thing is like we we kind of like I'd love to see more celebrations of the things we prevented rather than the you know like the postmortems are great um and the the fusion of funds all wonderful stuff but I don't think we celebrate enough the stuff that doesn't go wrong >> I agree yeah here in this talk basically and in the whole day you see all the biggest hacks discussion about how they happen and that's very useful for lessons learned and everything else but I think you're absolutely right we should actually also talk about like how did we prevent this hack uh unfortunately because in the world of automation and now in the world of AI we don't we we may not even know that we

prevented a hack. We just put some tools over there. We put some strategies on those tools. You know, we follow our operations or standard operating procedures and stuff like that. And potentially we blocked a lot of hacks that we never noticed that we actually blocked. So, >> so there's a lot of technology being built on top of that. Some of the guys are around here probably bitfinding.com uh they recently a couple of uh week ago maybe they they rescued like uh $1 million from from from the the balancer attack and uh really what they did and I think that's you know I think what they do is really really cool because they they keep scanning the the monitoring the chain whenever they see uh you know some contract that's being deployed that they detect that you know the there's

some some patterns some heristics there that implies that it's likely an exploit. it uh they start fuzzing it and uh they try to call the the the the smart contract before the attacker does so and that's how they actually took the funds and that's a very different way to do it but uh speaking about technology and monitoring onchain monitoring to prevent attacks would love to hear from you what's like what's your perspective on that >> so you know I think it's like an evolv an evolving landscape right it's um there's always like these new tact tactics, procedures, right? And so you have to stay really really up to date on kind of the latest and greatest threats that are happening across the ecosystem.

You know, I also think that it's it's um it's incredibly difficult to go in and I think it's it's it's easier to do this on like one chain for one ecosystem for one kind of thing and it it becomes incredibly hard to go and do this across a lot of different chains for a lot of different ecosystems for a lot of different types of companies. Um and uh and I think you know we're we're really lucky to have incredible partners that push us in the right direction, right? So, you know, we work with some of the largest protocols, we work with some of the largest chains, work with some of the largest, you know, uh, wallets and asset managers, and, you know, at the end of the day, they go and tell us what to build and and that's what we ultimately end up doing. Um, and so, and

so I think that, you know, I really like your guys's idea about like, you know, the the the panel next year should be, you know, let's talk about uh the billion dollars that wasn't stolen this year in crypto, right? Um because there there are good teams and there are you know we have great partners that do a lot of work to to ultimately prevent a lot of these different things right um we've seen things like you know after the buy bit hack right we have like a a newer product that basically you know verifies transactions before you sign them across various different MPC wallets or or or or multisigs and and you know we've seen tons and tons of teams adopting these things right um and uh and and you know basically telling us hey we we need this we want this and and then going and using them um as well as

you other kinds of solutions as well. And so I think ultimately like I I agree with you that that we need to we need to definitely highlight what the industry does well um in order for for that to percolate to other teams that probably do it poorly. >> I just feel like it won't get as many likes on on X. Like >> that's [laughter] a different problem for a different panel. That's the algorithm. So people talk about opensource being you know not secure you know it's open source it's out there an attacker can see it but at the same time people can just you know auditing so there is like this this kind of a con uh you know contrast between it being more secure or less secure how do you guys see it and

specific specifically when it comes to crypto projects >> I think in the long run security by obscurity never works Yeah. No, I agree. I like the idea that if you build a secure product, like most of the cryptography algorithms from years ago were always like open like you could read the entire algorithm, you could understand exactly how it works and you still couldn't break the codes, right? Because like like tripled like everybody in the cryptography uh kind of area, they they understood triple. You looked the whole thing, couldn't break it. It should be the same thing with our D5 protocols. you know the code should be open to everybody to read and there should be a lot of eyes on it which isn't in a lot of cases that that that's what happens right um but then I don't I

guess there is some argument for having closed source because if there is a vulnerability you wouldn't want to be out there I don't know if anyone's make case for that but I I I would prefer things being open >> I think there are areas where that definitely will benefit from being closed sourced or like private u I think you would you I still you know want to have things that being open source course and really public uh but then make public the inputs, make public the outputs. Maybe don't make public you know how the inputs you know work into outputs and you know just keep that little you know enclave if you want to call it in some way private. Uh but there's still if if you know if we start building closed source then we haven't made anything better in the world like we're just going back to web two it

>> you know it's ultimately like the Apple and Google debate right it's like uh you know Apple is security by obscurity you know Google is like fully open sourced Android right um what's more secure right what's what's the perception of what's more secure as someone who spent time you know like six years like researching vulnerabilities for mobile devices like it's it's freaking hard to find a vulnerability in in in in in in like fully patched Android. Um it's also hard to find in iOS for what it's worth, but you know, nothing's really closed source. Like if you have the binary like you you you can you know you can you can do a lot. Um and so and so I think like even on chain, right? Like if people start deploying closed source contracts or stuff like that, it's not really closed source. Like you you just like

have a harder time reading the source. >> Yeah. Yeah. Good point, >> guys. But let's face it, incidents can happen, right? So Joe, what what's been your experience in like responding to an incident specifically in the realm of crypto? Like what what you see that it's different when responding to an incident in crypto? Things happen extremely fast at you know at oftent times we don't have enough information. What's different and how do you guys have approached it? >> I I think one of the biggest points is that speed is much more important in crypto. It's still important in a web two compromise, but in in web 3, speed is so much more important because of that immutability of funds of of

transactions on chain, right? And so there has to be awareness, there has to be monitoring and folks need, you know, instant response plans, right? So many organizations don't have them. They think that they can just call up someone on you know incident happens they email someone they call up someone say hey I need you know someone to come do instant response and what if there's not availability and that happens right if there's no SLA no ser service level agreement there's no guarantee that someone is going to be there to answer the phone when you need help I would say that's one of the biggest differences is time matters much more >> that that is one thing I I encourage protocols to have is to have some sort of retainer with an organization to help

you with instant response because exactly to to Joe's point, if you like were to message Mandant and like set it to an emergency, say, "Hey, you know, we need you now." And they're like like, "Yeah, we're backed up." Like we'll we'll call you in a week and we'll send over the SLAs's to sign and we'll and then it's like, "Well, that's way too late." Um it's hard hard to justify like when I go to like my CTO and say, "Hey, I need to spend this money on a just in case we get hacked." um because he's like, "Well, we're not going to get hacked, right?" I'm like, "Well, I don't think so, but you know, if we do, we want to be able to call someone super fast and have them tracing funds and have them responding." And there are some good companies out there that do

that. Um you could DM either of us and we could like put you in touch with companies that are like there to help that sort of thing. >> I would I would say they're not that different to be honest. Web two versus Web 3 in incident response. And the problem is that we think that we need to execute incident response once the smart contract is hacked or you know or manipulated or something like that. But in the end having a security team having somebody that thinks like a security guy you know pessimistic and so on is going to help you because there are always indicators of compromise. always somebody is starting you know we see it in the polygon blockchain for example we see po people deploying you know contracts and hm they they've been uh you know uh funded by tornado cache and

then that's starting to be an indicator of compromise you start seeing people that are sending you messages asking you you know hey why does this function behave this way why does this function behave the other way there's always you know and that's what security teams do know they're monitoring the things and they're actually the ones that can tell you like actually this guy asking in this question may be malintentioned. You know, he may not be asking it because he's integrating. He's actually asking it because something like that. And you see that across a lot of the hacks. You end up when you start looking at the breadcrumbs of a hack, you start seeing, oh, this guy actually went to the forum and he started asking about this like a month ago and then you see that this same address was actually deploying contracts on a test net, you know, and

obviously you cannot see, you know, what the smart controller was doing, but you can decompile the bite code and stuff like that. So there's always little breadcrumbs like, you know, nobody's perfect and in the end this is all done by humans so far, uh, all the hacks. So, you know, you are always going to be able to start initiating your incident response process hopefully before the hack is is happening or at least keep that in mind. So, when the hack happens, you already know like, yeah, I I I I saw this coming. And yeah, >> there's something they do in in web 2 security that we did a long time ago that is I don't think it's totally made it into web 3 yet, which is the hunting aspect. So when you have a dedicated security team, when they have nothing else to do because well, first of all,

they're almost always busy. But uh when there is free time, you can start hunting for activity. So you're not waiting for alerts to come in. You're actually making hypothesis like, you know, maybe contracts funded by tornado cashache are more likely to be malicious. So let's go and find all this stuff, right? Let's go and look for all those emails, all those fishing emails that my employees are reporting. let's actually go through a bunch of them and see like how many of them are likely North Korean, how much are likely e-crime, how much are likely, you know, something else and who clicked on like if you don't have someone dedicated to security. That's just not going to happen. Like no one is there to look at that stuff >> and you're going to hire for that.

Sorry. >> I was going to say to build off what both of you are saying, everyone is aware of, you know, the defender's dilemma, right? You have to protect everything. If the attacker finds one thing, then it's it's all over. Well, there's also the attacker's dilemma, and this is what a lot of folks don't think about. If the attacker sets off one alarm and there's someone watching, then the gig is up, right? But there has to be someone monitoring and watching in order to see that alarm go off >> and put in the traps, put in the alarms, like you know, web two, we do a lot of these things, honeys and and those type of things. So you a lot of hacks can be preventable. You can actually get indicators. I was going to say talking about the threat hunting or you know red

teaming and things like that. You don't even need a security team for that. No, you can ask your own auditing company. Normally, I I think that most of you, if you ever hired an auditing company, and I see a lot of auditing companies over here, you probably get like, hey, here's a bunch of contracts. Uh, do your work. I don't know what it is, just do it, you know. But in the end, you can actually tell them like, please focus on these areas. No, maybe not for the first audit, but the second, the third, the fourth. You're actually going to tell them like I actually want you to focus on these functions because these are the ones that are most this is a critical path and this is what I want to cover.

And then the third audit you're going to go and like I actually I want you to start through this with these type of assumptions. No, think this way. Please plan you know your thread models in these areas and that's going to help you build a much better you know you know thread landscape as well as you know plans if some of these things happen. >> Right. So, what is your last message and uh probably the most important message that you'd like to leave to the founders out there? So, mine's really easy. If you do not have a security person and you take care of other people's money, go hire a security person. Like, you are holding on to other people's funds. You should just go out there. There's tons.

And if if anybody tries to say it's hard to hire security people, I'm in like this web 3 chat and several of them full of security people looking for jobs, right? I don't know where the disconnect is, but there are a lot of people out there in the security, especially in web 3, who would love to be offered a job. They just don't know who to talk to or how to apply or I don't know what the disconnect is. But anyways, go ahead, Joe. >> Uh the thing that I hope folks take away is awareness and education. I think that we need a lot more awareness and education around security for startups for founders and that will lead to a much safer ecosystem.

>> Uh I think I'll say uh security is not difficult nor expensive and I think that's a misconception. Uh you know with that security is nor security in web 3 is not different from security in web 2 like please go and ask. I think people are just afraid of asking ask for help. Like you can even get it for free. You can get audits for free. You can get a security review of like your website for free. There's a lot of people that are really willing to help. And at least, you know, in this panel, I think nobody here would say like, "No, I'm not going to help you." I'm sure I can find somebody that's going to help you with your problem in the budget that you have and in the time that you need it. So, don't be afraid of asking. Um, yeah.

>> Yeah. Maybe, um, I agree with everything you guys said. You guys, uh, I think super super super insightful. maybe that um security I think in this industry is not a cost center. It it can be a differentiator and when you know leadership looks at kind of security practitioners like to bring on and doing these things they should probably think about okay great maybe if I have a headed security I'll get more assets deployed in my protocol maybe if I leverage you know this kind of solution or that kind of solution to go and kind of bolster security I will actually be safer and I will actually increase TVL I will actually kind of you know generate more more revenue. Um I think it's especially true for financial infrastructure and especially true for for for the infrastructure that's being built across the industry.

>> The real cost center is getting compromised. >> The real that I mean that's company ending. >> All right. Thanks everybody. >> Thank you. [applause] Everything.