AI-Driven Prevention of Malicious Blockchain Transactions

Transcript

awesome good morning everyone thanks for coming instead of staying sleeping I really appreciate that I know it's hard because it's 9:00 a.m. but really appreciate it I'm Carlos I'm the head of AI at the fora foundation and I'm going to be explaining to you today how we are using AI to actually prevent malicious transactions and yeah in blockchain I know AI blockchain I'm a big fan of saying this keyword I promise there is only one more over the presentation but it's going to be fine so first of all I want to give you a little bit of context of because now I'm talking AI for prevention but fora started a couple of years ago with monitoring so we started

using machine learning to analyze blockchain activity and we were looking at um how a particular address was funded the smart contracts they were deploying and we were using machine learning in several different flavors to be able to detect when something was going to be attacked but we were doing monitoring right which means that we were finding that someone was potentially an attacker and we were saying you have an attacker here but that's it right and that's not so enticing it's not so interesting because yeah okay I know that's an attacker but I don't have any of these tools to actually stop this attack today we are moving a little bit forward and

instead of just using this previous part that we were commenting of we are looking at funding we are looking at the Smart contract that they are deploying we are just directly screening the transactions themselves to try to identify whether a transaction is actually performing an attack or not one of the things about detection especially in this last couple of years is a lot of people have been working on this thing and detection methods have been gotten much better and a lot of attacks that before we were not able to detect now we are able to detect before they actually incur into any losses right but we are able to detect does not mean that we are able to stop them and

even in cases when we can see an attack and we say like okay this is coming you you might even start losing some of the funds because like a couple of transactions already went through and you were at a bit too late plus there is the problem of there is not a very strong mechanism so what to do to actually stop these attack transactions right because you can stop your protocol but then it comes with a lot of other problems then people cannot do like normal operations and you are just like blocking completely and it's like a very extreme measure so we think that there is a better way of doing that and this is what we are introducing as fora fire wall which mostly what it's doing it's like it's looking at the transactions in

a from a stainless stateless point of view and this is something that I'm going to spend a of time afterwards because I think it's really interesting and it's really important in order to be able to stop transactions and this system is actually able to block these transactions before they are happening we are not talking about sping the the protocol we are not talking about further influence that literally just stopping the transaction itself that it's an attack transaction so there is no other problem to operations other than you are not losing your funds that sounds amazing right great so now let me give you some more information on I have stateless here I'm going to explain why I mean by stateless at this point and I'm going to

also explain a little bit to you how to create these AI systems and what to do with them what's the process to improve them how to validate them and so on and so forth so great by stateless now imagine that we are looking at a transaction and we are just looking at a transaction right and we want to identify whether that transaction is an attack or not like are will is someone is is it behaving in a different way how it's supposed to be and someone is gaining money or funds or whatever in an illicit way yes then we stop it easy right well not so much otherwise we would have already be doing it like and no one would lose anything but let's look at the transaction itself what do

we have there we have what we are using to generate this transaction which is like who is sending the transaction right where it is being sent and what is being sent so the data and the state of the chain like you need the block number just to have an idea of how things are there okay that's your input you have nothing else then what you can do is using that information since it's a terministic you can simulate your transaction and then on top of having the from the two and input data you would get the logs of so what actually happened in the transaction and the traces so who is interacting with who like who is sending like whatever delegate call to whoever other contract and know that

information and that's it that's all the information that we have right it's quite different to how a normal person right now a security researcher would go into a transaction identify okay this is an attack or this is not an attack we are just looking at the transaction and actually in my opinion this one has a little bit of a problem also because we don't necessarily want to stop people just because of who they are or who they interacted with to actually stop things right so just because you are a person that was interacting at some point with tornado cash doesn't mean that your what you are doing in in this particular moment is actually legit and you just want to be

completely Anonymous you don't have like any malicious intent so just the fact that you are funded by one of these mixers doesn't mean that necessarily you are going to have this illicit content right so the the fromom the two even though we are using the mass Flags because it's a lot of valuable information to know within the interaction who actually started the transaction right like we know that a certain address actually is the one that is doing this call to this other address but we don't have any extra context about that address we just know that this is the address that initiation that initiated the transaction and that's it we don't know anything about it so this is what we mean by stateless like each

transaction is stateless is completely independ dep on anything that has happened before and this is something very interesting because let's compare it to how a stateful or like how right now any security researcher would understand what is an attack you would see a transaction that is weird that you have like a lot of there are a lot of funds moving and so on and so forth and then you would go to the other state is starting you would check their funding you would check whether it's coming from a mixer whether they are like coming from other ilicit accounts like like how are they getting funded how are they getting their initial money what is their past Behavior right so are they doing also weird stuff do they have like a history of scamming

people before and it is known or do we have like any extra information about what this address is doing or like which smart contracts have we deployed we can analyze this smart contracts we can see whether they were good or bad they were potentially malicious if they have a story of deploying malicious smart contracts and were funded by tornado cash and how really scam people then well I guess this person is probably a scammer and attacker right so it's kind of dangerous but and then we also have like some heris rules of like okay you are completely emptying the pool blah blah blah that it's like either very general or very protocol specific so in the end what I want to say here is that this is a lot of extra information that we are taking into

consideration right and this is what we are considering by stateful detection because we are taking to state the previous state of everything that has happened and we need to have all this context for transactions and so on but having this context obviously helps right because it's always nice to have more information about it but it's a slow like whatever you do you have to have this things somewhere you you cannot have it like excl you cannot have like a huge memory with all the others and understanding what each one of those is doing so even if you have like a nice system where all of this is together and you still have to call it you are still adding like a couple hundred milliseconds for every transaction that you are trying to detect and when you want to go at a proper scale and you

want to go more to like when you want to be working on a rollup instead of just protocol specific that is just too slow you are going to potentially break the stuff and this is something that it's just not acceptable to be that slow when you're doing this thing so this is why doing stateless and just looking at the transaction itself it's really valuable right it has like these two main points of first of all full it faster because all the information is already there and you just need to run code over it plus there is the um I completely lost my line of thought whatever cool so now let's go to how we can train a machine learning

model up until now right this is a very standard machine learning life cycle of what you would do and here we are going to have to apologize me I'm going to use the word Model A lot and it's going to mean two different things because like you have the model of when you're modeling the data and the model of like the Deep learning model when you're doing inference I'm going to try to do deep learning model if I don't you can hate me afterwards you can't fight me for a coffee and should throw it in my face because it was confusing so great so how do we start we have the transaction right and we have like the from two data logs and traces and that's it the first thing that we need to do is model the data and that in this case model the data means

we check whatever input data that we have and we transform it in a way that it's it describes our transaction in a unique way and it's it has a format where it can be passed to a machine learning model in our case a deep learning model like deep learning models here and in this convers in this talk I'm talking about AI I'm not talking llms uh llms are not necessarily the best tool for this job but yeah it's a bigger umbrella there are other type of deep learning models which is what we're actually using just imagine what we have been seeing like some years back this computer vision we were identifying

things there the model of the data was the image itself because it's good enough but now what we need to do is transform it so just for the sake of the example let's let's imagine that we are modeling into a table which is an input that you can actually pass to a machine learning model so we can start with like a very simple model right and we start with something very simple and we will just assess and have it as a baseline for that we would create like a model that has again I'm going to be extremely simplistic but what was the total profit of the transaction so which address profited the most and how much did it profit and how many traces did the transaction have so did someone profit a lot was it a complex transaction that's like a extremely simplistic model of the

data but that's something that already allows us to keep moving into this into creating this machine learning model what is the next step we have to collect a data set we are going to use what is called supervised learning supervised learning mostly what it means is you present the model with a set of data and you are telling the model like hey you see this observation this is bad you see this observation this is good and you do that a lot of times very atively and in the end the model learns how to do over that thing so for that what we did is we create this data set of previous attacks of the transactions that we're actually subtracting the funds but not only you need to also have

like good transactions to be able to differentiate them right so you create like this big data set with these hundreds thousands of attacks and these thousands tens of thousands of banning transactions and then you train the model right training the model here and this here is the Deep learning model and not the data model uh what you do is like you do this iterative process over a certain amount of a I don't know how familiar with machine learning but this is like standard procedure you train you have a loss function and mostly the loss function what you're are doing is optimizing a function so that the model what you are telling the model to be true so these labels of the data set that we commented before it gets minimized and the model has learned what

you are actually teaching him okay now you have a train model with that you can already defend blockchain yeah sure you can try but then since we are like extremely simplistic point of time we have like a very simplistic data model it's not going to work that well so what do we do you go into evaluating the model and you take millions of transactions of normal operations of several blockchains and you evaluate that and you see that your model is bad because at this point it has it has not enough information to actually learn whether a transaction is good or bad right so then you go back to square one and it's like okay I know that it's bad how can I improve my my my

data model so you start thinking you take a look at your evaluation you look which transactions you fail a lot and then maybe you saw that there are transactions where someone has there are several people with profit and in those ones you were very bad okay then you add a new variable to your model data which is okay this this um this transaction actually has three people on profit and then you do it you do again in the collection data set you already have a data set from the previous step but you can use the most representative data points that you have um collected during your evaluation because that's the way in which you can keep improving your model

like you evaluate it you see where you fail you tell to the model you were wrong here and the model say thank you and it won't do that mistake again then you train the model go again to evaluate the model it's a little bit better you see another thing and then you keep doing this iterative process until eventually you have a model that actually you're happy with the performance and that it's actually able to detect attacks only using transactions and that's super nice and beautiful right this cycle is something that it's called mlops which is the other keyword that I was missing at the beginning it's like this is is the most keyword in ml after AI probably right now this is literally devops applied to ml so it's just a way of having like an itative way

of improving your models and monitoring them to make sure that they don't get worse over time because that's something that can happens and that you are still finding the stuff having this type of mlop Cycles is also extremely useful because imagine that you are working with some customers right and then there is a type of false positive that it appears often and here often means like once per month right but like once per month you have one false positive with one of your customers and it's always the same type of transaction when something particular happens your model gets confused and it says that it's an attack even though it's not okay doesn't matter we have the cycle we put it into our cycle the next iteration of the model is not going to be wrong at that point so it's it's a model that by

itself it's good it's already able to detect a lot of attacks but it's also going to keep learning and learning over time and it's going to learn about the things that we are actually tring so it's just getting better and better and the things that we actually care about what is the output of the model right because right now the previous slide was a little bit theoretical and was like yeah okay the output of the model what is what is this model actually giving us like in the example I was using before of pictures if we are using computer vision the output of their model is like is this picture a cat or a dog is is it a cat or is it a dog but here we are trying to evaluate how risk a transaction is right so the way that we Define the because this is

something that you have to Define in the model and is something that you need to Define when you are defining your loss function and your objective and this is your end goal what you want to do in a beautiful world is that every transaction that is an attack is at one every transaction that is not an attack is at zero of course that's never going to happen because there's something in the middle but that's the goal right so what we end up is like with this spum and the model gets good enough when most of the attacks are as close as want to possible and most of the no attacks are as close as possible to zero something that is also interesting is that since we are generating this risk score and this is not only like a single value right it gives us like a complete

interval of data we can actually depending on who is using it they can decide how risk averse they want to be because some people they want to be like no if you are even a little bit concerned that this might be an attack I want you to stop it then you can say like okay everything that is over 0.5 gone you are going to obviously have a lot more false positives because well um that's how things work in automatic systems but you can make sure that you are not losing anything because over 0.5 it's very extremely difficult that nothing is found on the other hand if you just want to make sure that it's a stop if it's like a very big attack and like the confidence is very high because you prefer losing a little bit of f instead of annoying your customers then

you just go over 0.8 which is recommended value that's why it's red and then you will only stop like attacks that are very obvious attacks so this is actually very interesting because it allows some configuration even though it's a little bit um esoteric at some point but you you can get like some understanding of how risk a transaction is and you can decide based on your risk aversion so now this was the detection part right like this is like when we were looking at the transaction um what we actually can say about the transaction being an attack or not but I also proom I also promise you that we are actually stopping the attacks and preventing them how do we do that I'm

going to go very low here like I'm not going to go into a lot of detail because the point of this talk was mostly to talk about the detection and the AI that we are using we have a workshop today at 3:30 where we are going to go more into detail into this otherwise you can find either Andy or me we are going to be around and we can explain a little bit better but the main idea is like this right so a user s a transaction and the main point is that for users don't need to know that they are already protected by firewall right like this has to be as easy as like users need to know that they are a little bit safer but they don't need to know how it works it has to be as easy as possible but it also needs to be very

easy for actually protocols or rollups that actually want to implement this thing so I promise you it's very easy and like we are making it easier every time we have a couple of different flavors of of how this thing is implemented depending of whether you are a rollup or whether you are a protocol um it works a little bit differently but like both of them are very low friction implementation and once it's set you can just set your threshold and any transaction all transactions are going to get screened and the transactions that are over threshold are going to be not executed um and it really works like this right so a user sense a trans a transaction it goes normal operation as

that user would be um the transaction gets screen at some point like it gets at going a little bit to for the firewall it gets analyzed we decide a risk score for the transaction based on our machine learning model then the transactions that are over the threshold that the user not the user like the protocol user actually defined are dropped because they were too risky and then they get log there is a dashboard they can take a look at it Etc and they can decide okay you were wrong here you can your your next model shouldn't be wrong in these type of things and the transactions that have low risk So Below the threshold they go normally um the

user doesn't see it because it's fast so it's very seamless and like most of transactions very most of them are never going to see any of these things so I'm talking Al a lot bit theoretically I want to show you some numbers of how this actually work right and like the results that we are having for with the model that we are having right now first of all like we did some test on September September had like around $40 million lost due to attacks out of those 40 Millions we detected 99.8% of the transaction that lost this money so almost all of the attacks that happened were there out of 40 Millions

we defended some we could have defended something like 39.9 million it's amazing because again it's just looking at the transaction we don't have any context we don't care how you were founded we just care what you actually promise that you're going to do like I think this is very important to take to take home that it's not about what you have done before it's exactly what you have already committed to doing the blockchain right now that you are being stopped so it's not about previous you can do whatever you want before what you are promising to do because it's the transaction that you are sending at this point is the thing that might stop your transaction so when you are you you are showing that you want to to steal those funds then regarding false positives at the rollup level so when we are

analyzing all the transactions over a chain we are having at like one false positive around every 150,000 trans transactions or so or better but like let's go one every 150,000 transactions this would be what um 10 F positives in the whole main net in a whole day like some other chains are like much better than this still so this is not a lot of transactions this is like for example arbitrum this is like almost 500,000 transactions we had one false positive which is the one that it's in the red part like this is the risk score that I show you before just like a histogram it's in logarithmic scale okay like the one on the right it's a one the one on the left is a

445,000 so if it was a logarithmic we wouldn't see the risk one so that's why it's in this way then for example in base we also have one that it's at one right yeah casually this was around the block when the summer money incident happened so that's the transaction that was in summer money so we the only actual risky transaction that we have and this is already around 100,000 transaction is the one that was actually an attack so this is something that we could have prevented and then it works a little bit different when it's for protocols because like they are more concentrating and things they are doing and like direct transactions are like easier to scan and so forth and here usually what

you can expect is like to have one false positive around every three months but let's be honest like defending 99.8% of the money for just the pain of once every three months having to log out into your dashboard and say like hey this was not a false positive that you know it's going to get improved over time it's very nice and this is like an example of one protocol that we are working with so just for you to know that there are literally nothing even in the yellow and orange zones so this is for theable this was how we did how we are using AI to actually start H stopping malicious transactions anything that you want to uh to ask us like Andy and I are going to be around today we

are going to be around in Devcon find us speak to us you can contact us by email whatever you find and yep I hope you liked it