ep 9 – Bridge Hack, Wallet Hack

Transcript

[Music] welcome to web 3 with a6 and z a show about building the next generation of the internet from the team at a6 and z crypto that includes me your host sonal choxy this show is for anyone whether developer engineer researcher artist community manager company leader entrepreneur policymaker or others seeking to understand and go deeper on all things crypto and web3 this week's all-new episode digs into recent high profile hacks that took place in the crypto space over the last week we not only dig into the details what happened including a more technical breakdown of the how and how we know but we also cover the categories and issues specific to and not specific to web3 security as

well as solutions and advice for builders we also touch on related trends and topics such as the role of open source communication around hacks as well as social media status signaling and much more throughout we try to help tease apart what's hype what's real as well as what signal and noise in the narrative out there joining me for that this week are experts from the asics and z crypto security team including security engineer matt gleason cto riaz faisula boy and ciso or chief information security officer nasim erikyak both of whom worked previously at facebook anchorage and docker and nasim or nas also appeared on an earlier episode of the show on nfts and security in case you missed it but for this episode just to quickly recap for your

context the hacks that we're specifically covering are first the hack of the nomad bridge which connects several different blockchains including avalanche ethereum evmos moonbeam and others with reported range of between 185 to 190 million dollars stolen and then we also cover the hack of the slope wallet a non-custodial browser-based wallet that was reported to affect nearly 8 000 users on solana as well as other ecosystems with a reported range of between 4.5 to 8 million dollars stolen it occurred a week ago and slope just posted their latest update today confirming some of the details we discussed in this episode which was recorded a few days earlier as a reminder none of the following is investment business tax or legal advice

please see a60.com disclosures for more important information including a link to a list of our investments with that the first voice you'll hear is riaz followed by mats talking about the nomad hack and the broader category of d5 bridge hacks [Music] this one to me is interesting because when we think about web3 security the impression that i get is that many people think of it as a very obscure dark arts sorcery a smart contract yes and how blockchain and web3 works totally and when you step back on on this incident it's really a logic flaw about updating state one place and not how to handle it in another place nothing specific to

how bridges or validators key management any of that stuff is nothing unique to that we've seen similar hacks before both in web 3 and web 2 and earlier systems and so i think we can take a lot away from just software development life cycle how we test upgrades how we think and reason about sensitive code but it's one of those those hacks where everyone is again zeroed in on bridge hacks are bridges a bad pattern are they all insecure are they all the big targets and in each of these incidents i'd say there have been different classes of attacks some of them are specific to smart contracts and very specific whether it's oracles or other web3 native

concepts but a bunch of them haven't and this is an example of one that really is quite a well understood type of of bug but obviously affected a bridge yeah just to give people quick context just at a very basic conceptual level bridges basically connect blockchains they're exactly what they sound like they're a bridge literally between two different blockchains two different blockchain ecosystems which is really important because that allows you to connect different types of information transfer assets and tokens more importantly enable you know cross-chain collaboration etc so that is a point of a bridge and when i think of a bridge i live in san francisco you know and i'm constantly going to the east bay to visit my friends there's a lot of traffic on bridges because there's a lot

of people trying to get between things i mean i don't know if there's necessarily a lot of traffic on you know these blockchain bridges but the idea being that there is a concentration of things which would be my guess for why people are targeting bridges i don't know what you guys think about why people are targeting bridges i mean the key things to capture about why bridges would be targeted are twofold one that you already brought up which is that they'll have some concentration of funds this bridge had like 150 or 180 million dollars in it and the other one is they're built to transfer the funds so like the main logic in the bridge is to transfer the fund to someone else and so if you're going to have a mistake and the mistake is in the logic of the

transferring then you're going to allow you to transfer it somewhere else and that's exactly kind of what happened here right there's a mistake in the transfer process and they have a big concentration in funds and so the exploit that happens is i get to get all the funds or in this case they didn't get all of them but like they get a sizable portion of it and then cause a lot of chaos along the way i was going to add like you mentioned the concentration you know the cars on the bridge one piece of bridges too which makes them a target is much how you can see the number of cars and the traffic on a bridge from far away because you can go on your favorite blockchain explorer and check out these contracts check out what's going on out of a bridge you can see kind of the

equivalent of the traffic on the bridge and the funds locked in a bridge and you can measure your output of what you can get exactly and and similarly you have also full visibility into incidents as they happen like we started seeing a lot of different people that realized what the issue was and started also you know draining the funds some of them returned them and as part of the fun recovery process people who are essentially white hats but some other people may have just drained it for their own benefit and and all that thanks to the transparency of the system yeah exactly it's a constant theme right the very transparency is sometimes the very thing that it's a double-edged sword but i have to say it reminds me of like the

olden days of like highway robbery i think it's the first bank run style exploit we've ever seen which is just this like chaotic everyone piling on i mean the hack was a mess of transactions i was literally thinking that the analogy that i would use here is like looting like when you have looting and people are like all taking advantage well some some of those people returned the funds but like yeah it was is crazy so let's talk more about the technical aspects like break it down i'm curious for you guys to share how you know things as some of the method behind the scenes as well as the what happened so i'll start like really high level so this is literally a hack we've seen before cue bridge was kind of exactly the same pattern

when we look at it now so basically what happens is a contract gets audited it gets changed they set some weird insecure defaults at a weird path and then like it gets exploited like real real hard in terms of the technical details of this one so um inside the replica so this is a contract inside the uh their system there's this process function and this process function is going to go through and basically verify that like hey i have seen this message before it has been proven meaning like it's gone through a merkle proof so i know it's genuine and then i'm going to process what it tells me to do

and so what basically happened here is in the process of proving it they're using something called this acceptable route and the acceptable roots gonna go through and basically go okay has it been proven before um yes okay then it is that is proven has it been used before if the answer is yes then i'm not gonna treat it as proven and then otherwise i'm going to look to see when the merkle root that i'm relying on was put into place so a merkle root allows you to basically prove that like hey i've i've done this anyway so through all of that the bypass ends up being in some of the weirder logic

around which acceptable route is checked so in solidity when you call something when you call a mapping specifically and you call it with an uninitialized value it's going to just return zero and so what would happen here is it would go okay well what is zero an acceptable route in the case where like you gave it a message it is never seen and the answer the contract ended up giving is yes which should blow your mind because what that means effectively is if it's a message it's never seen before meaning like if it's a message that like has an address modified to something different or some something similar then

it's probably going to accept it and when it accepts it it's going to do what the message says in this case transfer two hundred 000 to me please and like the reason this happened is because a default route basically an initial route that was set to zero and so again the system thinks hey the root of zero is accurate and as a result i will accept any message that i haven't seen before which from a technical aspect is kind of scary right because what it means is you're a bouncer at a door and you're you're taking tickets doing what the tickets say but if you've never seen the ticket before whatever's written on it go do it that's not a secure system so it creates

a lot of problems and so that's kind of what we saw is this just like piling on of people realizing hey i can change the address in this message which by the way makes the message something it's never seen before but now i can send this and it'll give me 200 000 uh they didn't even have to know how the exploit worked okay so that's why the bridge was able to be drained to fund so quickly but i just want a bit more detail on the root cause because like how did that happen in the first place could you guys break that down a bit more so the root cause is essentially at the message passing level you have two chains for every single deposit that happens on chain a

the information needs to be transmitted to chain b and on chain b the funds should be able to be withdrawn at the same amount right what happened is that the message passing in the mechanism by which the messages get validated on chain b ended up being broken and this mechanism was broken through a software update that essentially did not account for the state that was stored in the smart contract and pretty much started taking the same variable in the state of the contract and interpret it

differently essentially the zero that was saying that transactions should be re-verified or were already included didn't mean the same thing after the same zero meant this transaction has not been verified yet approve it so this is the moment where a zero in the state you know a specific variable doesn't mean the same thing before and after the software update interesting and so as a company managing the bridge or as a set of developers managing the protocol you should obviously make the state and the code work together right when you update one you should update the the other but in that case only one

was updated and so this is really what happened during this hack i want to know how you guys like figured it out looked at it like you know how do we know what's happening realistically we basically start from what is actually going on on the blockchain and so if you looked during the attack which was like highly chaotic you see a bunch of these transactions and all of them have this call data that looks almost identical with exception of one field and that one field is the address of the sender themselves and so you see a few of these and you go what's going on like they're just sending the same message with just a slight deviation which is who to send the funds to

and then it's sending the funds to themselves and so when you're when you're sitting in that situation you have to consider well where could this have gone wrong and if you look at their system and you go okay they're going to take messages they're going to prove the message is real or genuine and then they're going to do what the message says where is the problem and it could be a problem of like doing what the message says or it could be a problem of how they're proving the messages and so what seems more likely in that case when you're just looking at it from the outside is they're probably just not proving it properly and so once you know that you go okay

something's wrong in the proof function and then you go looking through their functionality typically on like ether scan and then you go alright where could this have gone wrong and so you go look at the the process functionality you see that the only thing doing the proving is this acceptable root function you look into the acceptable root function and you're like man it looks okay-ish but it's clearly malfunctioning for some reason and then you you consider hey if this is a message that's never seen what's the acceptable route or what what root is it checking and you're like zero it's like all right what is the contract recording for acceptable root of zero oh it's a

one that's really really bad it's like yep and so once you get to that point you're like well i know there's a problem with proving i don't know if it's a problem for sure but i know there's a problem with the proving of the transactions and i can kind of surmise that hey people sending a bunch of transactions there must be a problem with proof i have at least one problem with the proof i'm pretty sure this is what it is now it doesn't have to be you can be wrong but you have a pretty good lead there and then you just kind of follow that one up and go okay what else could it be or is there something else i should be keeping an eye out for and in this case there's not really anywhere else it should be

malfunctioning and so that's kind of the process behind how you end up finding or how you end up like attributing this kind of exploit to like a specific book and a lot of that is instinctual at the end of the day because like it's a threat model it's like if you've hacked systems before you go oh this is where i think it goes yeah i think one thing with the bridges and just generally with blockchain is that we have access to all this transaction data of legit transactions going through the bridge about form transactions and we can use those as data points to help influence during this process which paths do we think are still possible or not possible and kind of dig down deep into this process so in this case we

could see the bridge transactions going through and so we knew that transactions were being processed and help us lead to the conclusion that maybe the processing itself is flawed and how it lets messages go through on that note is there anything to be said here about what would improve because i do want to add obviously not just what happened how it happened methodological but um suggestions for future avoidance of such yes for sure so across the what three security incidents that we've observed over the past years is that a lot of entities do not leverage integration testings as much as they should they're not

really building the systems that prevent them from making mistakes systematically the idea behind the integration test is that you're actually testing the behavior of the system as a whole provided some inputs and verifying that the outputs match what you expect for that especially when several components are working together aka code and state and sometimes multiple software components and so one thing that could have avoided this issue and a lot of the others that we see in the space is just being a lot more rigorous on the integration testing that is being done if you start first testing against a local copy of uh essentially like ethereum or if you're

testing on testnet or you know some other test environment and that you actually run the same code as in production in the same state as in production aka mainnet and essentially go through the process of ensuring that transactions are only approved if they're supposed to be approved that transactions only get applied once and cannot be replayed essentially this set of use cases that apply to your infrastructure not having these integration tests in place is pretty much flame blind you're essentially thinking like okay i trust my code because it is good because it was reviewed by some people internally

in that case not even auditors the auditors did their job before this code change and so you're pretty much relying on one person sometimes two people to actually validate that the entire system works what you should do is actually building software infrastructure that will tell you when you made mistakes that will prevent you from making these mistakes systematically so that's something that should be worked on a lot more across the entire what three space yeah there's a couple things here in the highlight one is writing tests integration tests with smart contracts is well understood

and people in different projects do it the other piece on this hack in particular that is interesting to me because it's again it's not new to web3 is understanding how like in this case we had a variable that wasn't defined solidity default to a value of zero and that's not new to solidity other languages go etc have default values and knowing that having tests that on your end check for what happens if this is zero what happens if it's undefined what happens is a very large value you have these this gut instinct of checking the boundaries of what can and cannot or should not happen but even then having this negative case in that if we have to

make a mistake and we let something go undefined how does the system handle and recover in that case so one thing about testing it's common practice to think about the various edge cases what happens if a variable is zero what if it's undefined what if it's a very large value negative and thinking through the boundaries of these edge cases of what you may not expect but it's still possible is a way to have these negative tests and these guard rails to ensure that your system will recover in the scenario that you have a bug or somehow some user input happens to be outside of your expectation are there tools readily available are we at a state here where there is tooling ready made i'm wondering if there's like an easy way to

do this if i were to add anything to the is there a framework to help avoid this the answer is no because the same kind of exploit is happening like multiple times to multiple organizations like obviously if they had the tooling to put them in a place where they could avoid it i think every developer would opt to avoid it and so realistically i saw this in web 2 space as well like it's just a very very hard problem because you cannot anticipate everything that every edge case that can happen and in fact even your security engineers are not writing like those types of tests because like oftentimes you really just need to run

them once or twice and then you're pretty sure that it's not an issue a lot of this comes down to like whether or not we end up getting a framework that like enforces some like quote unquote secure by default capacity so this is like what we've seen in industry ending vulnerabilities generally it's like hey we have a real problem with like cross-site scripting which is a web to vuln then you you have the design of frameworks that like don't allow you to do that on accident like you have to do it on purpose now right and the trade-off there is what is it like an innovation one like you know the ability to flexibly figure things out because i do remember this evolution of security i remember with networking

and other security it was very interesting to see them move towards a castle model to another type of model and it's very connected to how you can build things exactly it's like much more restrictive and so unless you have this very feature-rich framework that also offers this security guarantee it's a really hard sell yeah people just want to do what they can to make their product work and to make their product good for their consumers they're not going to care as much about the security guarantees of the the framework i mean web 3 might but like web 2 didn't and so like it's only when the feature richness gets to a point where we we need to do this because it's like not only easier

it ends up being better and then we just happenstance get all these little guarantees too yeah that's probably where we're gonna see that or what the evolution we'll see in this space as well yeah because currently like everyone is pretty much testing so there is the web3 frameworks across javascript and python and forge can also be used fortune foundry can can also be used to write unit testing and help in integration testing but the main id here is the one that matt touched on which is you need to share components build shared components that will become secure by default that will become the standard in order to essentially eliminate the class as a whole

and that might look like for example for bridges the message passing and transaction proving mechanism like it doesn't feel like the big category of bridges that uh verify through microproofs will gain a lot by re-implementing it from scratch every single time it might make sense to actually share it and try to build the flexibility into what kind of messages you allow to pass and so on as part of this framework but just like standardizing at least a higher level these mechanisms will help getting everyone to participate in the development and also the security of

these specific components right this goes to your point about open source too more broadly exactly give me a super high level score card on each one of these on the scale of one to ten and i know it's not like easy to distill i'm just trying to get a sense like where would you put that on like of all the hacks so far in the 3 space and also relative to web 2 hacks you'd have to give me a simple answer but give me the bottom line on that basically i would say that the nomad hack was pretty bad in the sense that it happened 30 in the open there was no real communication with the team during the entire hack the size obviously of the hack from just a financial perspective is uh is gigantic we're talking about uh close to 200 million dollars

and there was also no pausing mechanism as part of the bridge so as soon as the issue was found there was no way to stop it so it was kind of like on the white hats to gather as much funds as possible proactively before they were getting stolen because once it was found there is no coming back essentially yep it's a very large number we've seen larger number for dollars hacked but given that it was the entirety of the bridge could get drained and there was no pause as nasa mentioned was catastrophic yeah i'd say generally like a 9 out of 10. okay so just to recap we've talked about what happened in the nomad bridge hack what we know how we know it how to figure stuff out what to do but one thing i also like to do is sort of provide a map and a train

for the listeners for how to think about these things like where does it sort of fit category wise you know in the overall related web 3 hacks out there yeah on the category side this one was pretty much like logic bug and mismatching between state and code there was wormhole which was like more of a cascading kind of like supply chain slash security notice and then the ronin bridge hack which is more the validator slash key management level i think that like it's kind of like the three big categories that we see all right so that was a nomad hack which happened this past week a bridge hack now let's go into the other big hack that happened it's like almost exactly a

week ago which was a wallet hack of the slope wallet let's talk about what happened there and let's especially tease apart you know the fact versus the speculation because i actually found it really hard to follow this one and track it and then we can go into how we know in other details and big picture themes but first let's start with the facts at 4 pm pacific on wednesday a bunch of transactions started showing up on solana signed by the addresses themselves transferring all their money and some of their tokens out of the addresses it actually repeated i think one or two times so people who like transferred more money in got looted a second time in a lot of cases and effectively no one knew where any of it was coming from obviously like

the immediate suspicion is hey if so many private keys are getting hacked this is probably not fishing because we've never seen phishing uh be this successful um it's probably has something to do with wallets which i think is why people honed in on like hey we think the wallets are having trouble like we don't know who but like we probably should try to get that information and unfortunately user generated telemetry is obviously a little bit flawed which is that people will report what they remember or what they think the situation is and not what it is which makes it very difficult to respond to the incident so in the case of what we know now is

there were logs sent by the mobile applications of slope and possibly the web applications of slope to a century server controlled by slope that contained both the seed phrases and the private keys of wallets and these would be wallets attached to slope and so what it affects obviously mostly solana i think there were in a total of something like 50 transactions in ethereum but 50 transactions relative to 8 000 is kind of a tiny number and that's kind of like where it sat according to the facts right so this is not a speculating this is just

what we saw and what we saw unfold now if we want to speculate the speculation sits on that we can probably attribute the the hack to slope but that's still something that will have to be verified as time goes on and realistically we don't know if it can be completely verified and can you guys walk me through a bit more about what you saw as well like as this unfolded kind of how you know what you know right for this hack over 8 000 up to 10 000 wallets were compromised and drained in quick succession as we watched it unfold we saw the set of withdrawals from a huge amount of wallets thousands of different addresses and that all had

very different patterns withdrawing to four different addresses the addresses that were getting trained didn't seem to have anything in common people were not necessarily in front of their computers or or phones and the transactions were actually signed by the the signing keys of these addresses and this is a very different from the majority of the issues that we see in the space because what tends to happen as kind of like larger scale scams is you know someone puts up like a fake mint page uh the mint action when you connect your wallet and kind of like approve the transaction it it is actually delegating the

permission to drain to another wallet to drain nfts to join funds and more rarely is it like an actual withdrawal right so that would mean in that case that either someone managed to scam and get people to approve withdraws across thousands of addresses all at the same time which ended up not being very realistic because the owners of these addresses were actually not signing anything at the time that were not even at their computer or phone or somehow someone got access to the

cryptographic materials that are the private keys behind the accounts and there were all kinds of reports where the pieces of software that seemed to be affected if we were listening to the people's early report where that either all of the wallets are affected or solana blockchain is affected or the cryptographic algorithm that was used to to generate the keys was uh broken essentially all of these options were scary and so it took a quite a lot of surveying among users to understand exactly who was among the drained and victims was using which kind of software and

hardware actually and the answer lied in the fact that it was one given wallet or at least that seems to be the accepted option and the accepted boost cause is that the slope wallet was actually logging private keys upon key generation and setting them in some cases to a remote server a remote self-hosted instance that they own and so the interesting part here is that people were actually exporting their keys that they generated in the slope wallet into other wallets and so this is why we ended up having

the reports of people saying i was actually using phantom and got hacked i was actually using this other trust wallet and i got hacked and this is actually the part that was the most confusing and so we had a lot of very conflicting information essentially boiled down to complete loss of private keys because the transactions were being signed by the keys themselves due to the keys being leaked by in plain text by a wallet that was used for many users and in some cases used in addition to other wallets where they imported those keys or exported those keys to and from slope so we saw a multitude of users would use slope as well as other wallets drained over the the time frame of this attack

one quick thing on this why is it so hard to know in this case just really basically straightforwardly i mean i feel like if everything is kind of open and out there and we have all these security people that are so smart and all this tooling and a whole bunch of people why is it so hard to figure out what happened that's a great question i think there's multiple angles to it and this this incident in particular exacerbated the inefficiencies of how we collect and analyze this data so firstly the number of wallets in this case that were affected the number of users back to matt's point about telemetry and getting good reliable correct data about what actually happened as opposed to what people remember so that we can

then take those facts and move forward in this incident the data was not very clean and so it led to many different directions of exploration rampant speculation on twitter about even if solana itself was hacked at the network level if there were cryptography non-reuse bugs if there were supply chain issues across multiple wallets all of these being very unlikely cases but just given the data that was out there was very difficult to hone in on a path forward and in the case of this this incident two yes there is available data on the blockchain we can see the transactions we can see the wallets being drained but

there's also stuff we can't see so for example we can't see metadata or code of wallets if they're closed source or even for live traffic and what goes back to their servers separate from the blockchain itself and so in that case that's where we rely on incident response and collaborating with partners to really dig in and get that information and analyze it to move forward yeah and in this case the parties that are affected like the people with all of the information will not share it they're actively disincentivized from doing so because it could result in them getting sued over some of this and so they're not going to be in a place where they should

be that forthcoming about information on exactly what happened they'd go through lawyers and that's unfortunate for the ecosystem but is kind of like what they should do and so like we end up in this like speculative bubble now where we're saying like hey we think it was slope we think it's like related to this logging that we have evidence of but we don't really know right yeah the information before it kind of went dark is that autosec who worked with them on auditing and incidence responses that only a subset of the private keys to accounts that were trained were locked and so it is a bit hard to understand

the truth the damage control part or you know if there was actually more to this hack than just this logging from the slope mobile application okay that was helpful for answering that so what could have been done in this case theoretically again we don't know so we can't map it perfectly but what are some of the best practices or lessons learned here i would say that there are two things that i think we should work on as an ecosystem the first one is just having a formal process for incident response whether we're a bridge whether we're implementing a protocol whether we're implementing a wallet there is a lot of value in defining a clear process to gather the information get

the right person and not too many people in the right room right because that becomes really chaotic when more than the required parties are present and having the right set of almost like kind of like a checklist style approach to the incidence response this has been done for decades in web 2 and there is no reason that what 3 cannot reuse the exact same processes especially on something like wallets which is very much a traditional piece of software infrastructure right it is just key management essentially at the end of the day that ends up signing transactions and just like managing keys

so i think that there is a lot of value in working on such process and the other thing that i would like to see is just shared components as well as i was talking about for the bridge hacks i would like to see shared components on things such as key management i would like for all the parties involved in the ecosystem all the wallets to work together on the key generation to work together on building abstractions that make these types of errors logging you know logging keys essentially not possible key leakage should be a common problem to every

single wallet and it is not really gonna make your business but it's certainly gonna break it if you don't handle it well which is what's happening now and so truly working as an ecosystem to build these abstractions that enforce the secure behaviors by default is extremely important got it okay second hack quick score card and uh assessment how bad was it this one i'd say when we were first investigating looked like it could have been much much worse in terms of amount of funds taken because the number of wallets that were compromised was so large so that respect it was scary in terms of number of wallets being drained the funds actually

trained could have been much worse so in that in that sense i'd say it's lower yeah i would say that the same the exact same uh security issue with another wallet with metamask with some high profile and widely used wallet could have resulted in a 10 out of 10 for sure i mean it feels like a six or seven out of ten not because of the phone or because of what what we think went wrong that is catastrophic honestly for slope i feel for them like obviously breaches and anything like this is like the number one nightmare that anyone can have when you're developing these kinds of tools but just the general blast radius wasn't that

wide eight thousand wallets is not two million that's a very helpful framing okay so i got your scorecards on each you know putting it in context how bad is it really i do want to pick up one more thread you guys mentioned with this one about the narrative and the confusion like how off you know or confusing was the chatter out there for each hack like both hacks relative to the facts like what was the signal to noise proportion for each and you can answer this lightning round style if you want let's start with the first hack matt i think people found the root cause fairly quickly or at least what we believe was the root cause and so i don't think there was much noise there's a bit of noise on like not knowing what was happening during but i didn't see a lot of speculation on

like hey i think this wrong thing is happening it was mostly a bunch of people sitting down trying to find what happened we found what happened a few posts blasted out on twitter and basically told you exactly what happened so the signal the noise was very very good good ratio there all right and how about the second one the second one was probably the the worst ratio that i've seen in the space there are multiple reasons for this the first one being that a lot of the information is uh close source and off-chain so this is not something that the security researchers uh have access to the information that was received from surveying people was also very bad

and a lot of very incorrect statements so we got wrong data that we kind of like analyzed not as well as we should have and then people who were on the outside were also kind of making wild guesses that ended up being uh untrue the second part was that there was no guidance there was no one really in charge of the investigation which resulted in people going in very different directions so that was kind of like that was chaotic to the incidence response and research for the nomad hack people were making suppositions or incorrect data on when the audit was done compared to

the vulnerability being introduced and so it can actually hurt the reputation of audit firms involved quite badly and so i think that getting these facts right before saying you know oh such a such company had audited it uh how could that miss this and so on is very very important like you should really get your facts straight before um saying such thing because we're talking about like actual businesses that are based almost entirely on reputation yeah okay so just one last thread to pull on this one i just want to talk a little bit more about the social media side that you both commented on i mean there's clearly more nuance and implications here beyond the obvious can

you guys say more about that we had a lot of user presented data we had a lot of that kind of thing and then we had a lot of security kind of influencers who are newer to security and especially are more concentrated in the apsec domain trying to speculate what happened and because of their expertise obviously they're going to bias towards very specific types of techniques and very specific types of hacks and they're not going to look at some of the more kind of reliable things that we see that are really kind of boring right but like it boils down to um this is a very new space um we have a lot of people who have come in and this is like all they've kind of

known they tend to hyper focus on the things they find both interesting and the things they kind of know the most about as opposed to fixating on kind of the more boring stuff that we actually see happen and so the signal in noise was really really bad um and i think it continues actually to be pretty bad well that's the narrative on slope actually it's much simpler than people made it out to be and so like the speculation was rampant and so yeah we're gonna have a lot of security theater it's a little performative actually isn't it like it's a performative hacking as you know there's this phenomenon on twitter where pete there's like a lot of people performatively trying to like figure

things out and share and blah blah blah and it's become like like a social game and even on reddit people always try to solve problems i get it and if there's one thing i've learned covering this space a long time and i've heard this over and over and over again from every security researcher i've ever worked with which is this almost always the most mundane thing i mean you kind of said that earlier but it really is shocking how sometimes it's not actually the most entertaining thing yeah i mean like 99 times out of 100 it's the stupid boring thing and you're just sitting there like oh occam's razor always the case one line we had in media when i worked at wired and covered a lot of this stuff is attribution is hard you have to be very thoughtful i mean even on the podcast darknet diaries

they won't cover a hack until like way like at least a year later to make sure they have all the information they need before they go into detail because it's like really hard to figure that stuff out and it actually has implications on what people pay attention to what signals they inflate they add more weight to x it causes more distraction that's actually a real problem and like there is a big problem at least in the web3 space where um you're letting the security influencers kind of hit the narrative in the war room and so like it's a bunch of people that are frankly like hey i've been hacking or doing ctfs but like have you ever hacked a network and it's like no okay so like how would you do it it's amazing a lot of it becomes this like

the more infeasible stuff it makes it exceedingly hard to react to an issue and it also makes it very very difficult to recover in terms of like a pr aspect because like everyone's minds are gonna go to the rampant amount of noise and rampant amount of speculation that was taking place and people are going to be like hey we want to see that this won't be a problem in the future regardless of whether or not it was ever a problem and so we're going to have a lot of fallout for a lot of the wallets and them putting in efforts that are probably not warranted or cost effective simply because there's so much hubbub about specific types of attacks that

might have happened that are just super far-fetched and what probably actually happened is like exactly what happened to like four or five other places yeah totally i know we already covered what could help in each of those hacks but is there anything else we should be thinking about more broadly for next steps like any remaining or quick advice or mindsets for builders to consider that you guys want to share i think especially understanding that the asymmetry of the groups out there that are trying to get private keys valuable information valuable assets versus eight person small team or up-and-coming startup really understanding that security is so key to

the system but also finding ways to not be directly liable or you know associated with targets of high value because the asymmetry between who's out there and those kinds of attacker groups it's a very it's an uphill battle there's a lot more importance for web3 companies to essentially make sure that their company doesn't have anything worth stealing and so what you need to do is make sure that you don't have user private keys you don't have your own private key stored in like insecure places just make it so that if someone were to break

in they can't really steal anything worth anything is really the biggest kind of lesson learned generally in the web3 space from a lot of the hacks that we've seen not just not just these one note on that that doesn't make sense to me simply because isn't the point when you're a builder to have things of value like build things of value yeah so you build things of value in that you build platforms for users and things for people to like gain value from um when i'm talking about nothing of value hackers want basically the currencies right they want like the thing that is immediately valuable to them they don't want to like pirate your platform that would be a lot of work they want your users keys they

want your own keys they just they want things that are attached immediately to cryptocurrency which they can translate immediately into value that's why they're in the space that's why they're headed after these companies so when i say value that's what i mean i don't mean your source code or any of those kinds of things right that's very helpful the mindset to have is very interesting because it really clashes with the kind of wagami transparency and trust in the the the web3 space right it's we're all going to make it together we're all going to build the space together which is true but at the time of building that protocol the platform the bridge the wallet whatever piece of infrastructure

you're building you have to assume that your machine will get compromised that someone will open the door by mistake to the attackers and so trying to build with the assumption in mind that the bad people are already in the network they're already on my computer what can they do this is really the approach that you should take what i'm hearing you say when you say that nas is that it's it's like sure the default ethos could be very collaborative and friendly and we're all going to make it wag me but the flip side of it is your default mindset should be beyond defense always exactly and proactively be on on protecting as well makes a lot of sense and just one quick note on the asymmetry thread that both matt and riaz brought up

summing it up very simply most of the builders are much smaller and less organized by default than any organized groups of hackers will be it's a classic this has been true since the beginning of all kinds of acts so one note on that is not is it true though that in this context the only way to get around that asymmetry is to actually band together and this goes to what you said earlier about like having you know shared components and things that are solid because the community has come come around standards and ways to do it and obviously when you think of open source that is a key key feature not bug of web3 ethos and and when i think of open source i remember the classic bill joy quote about how the smartest people in the world will not only work for you and the benefit of open source here is

that you can have multiple people collaborating together to really harden these systems this is exactly the the idea essentially this space needs to come together on security critical components that can be reused across platforms rather easily and that do not make businesses but can actually break them when done poorly there are a lot of independent security researchers that are extremely interested in participating to these efforts we're actually seeing a lot of the security manpower being external to the startups so in order to leverage them we should probably open things up

build all together collaboratively so that we don't even need to think about tomorrow hey did we mess up this thing when re-implementing it right it's a little bit like people building on top of linux you can create you know your own distribution of linux tomorrow but you're not gonna have to rebuild the entire kernel from the ground up you're not gonna have to think okay like did i re-implement the linux kernel the right way did i interface properly with key management and and uh and hardware and so on and so on so we've done that for the past decades we should continue to do that in web 3.

riaz and i have been working on very extremely sensitive infrastructure for the past five years and this is truly the thing that we realize is is needed building these common obstructions along with the rest of the industry that can be reused and eliminate the class as a whole forever right composable security this is really a rising tide lifts all boat situation where we can build on top of known good secure libraries tools so that we don't have situations where people can easily shoot themselves in the foot make mistakes as an ecosystem and i think really if we band together as a community

and work together on this i expect it'll help us build the next generation of amazing web 3 apps that's a great note to end on riaz nas and matt thank you for joining this week's episode of web3 with a6 and z thank you so much thank you bye thank you for listening to web 3 with a6 and z you can find show notes with links to resources books or papers discussed transcripts and more at asics and see crypto dot com this episode was produced and edited by sonal taxi that's me this episode was technically edited by our audio editor seven morris credit also to moonshot design for the art and all thanks to support from asics and zcrypto

to follow more of our work and get updates resources from us and from others be sure to subscribe to our web3 weekly newsletter you can find it on our website at asicscrypto.com thank you for listening and for subscribing let's go [Music] you