EVM.lua: Supercharging Ethereum calls with in-memory execution

Transcript

All right, good morning everybody. Uh thank you all for coming. Uh you know it's a it's a wonderful event and wonderful community as well. Uh so I came here today to talk about this project. Uh it is, you know, my pet project that I'm actually obsessed with and it is um to try to solve a long-standing problem regarding RPC node scaling and it's called EVMU. So my name is Everton. I work for AWS, but before that I spent almost five years at the Ethereum Foundation. So I worked on the um mist project, the wallet um ethereumjs u and a few other projects internally including defcon

website and blog. So you know doing a little bit of everything uh we learn a lot and actually you know working for AWS now some folks might wonder well why AWS right actually it kind of gives us this interesting opportunity to work and do stuff like this right so to take you know Ethereum data sets make them publicly available and folks can download and run for free they can query using Athena it's running since 2022 not a lot of people know that we don't make a lot of noise.

Um also we we have you know uh AWS uh programs for startups web three startups that we're actually partners with the Ethereum foundation and the EF if you're a grantee you can have AWS credits to build a company there right so that's the type of interesting work that we kind of unlock once we have you know somebody that came from deep ecosystem to a cloud company like that that's super powerful and also stuff like this this is not created by me but it is from the same team. So we have a team of other experts and they're building you know just a lot of acceleration to run nodes of you know 14 protocols and and on.

All right but back to business. So this talk is really about challenges on RPC scaling and a problem that that I can of course observe from the customers that run uh their web three workloads uh on AWS. So we have the privilege of working with Coinbase, Binance, Crypto.com, but also a lot of DeFi folks, right? So some of them I can even name like one inch, right? Those folks are on AWS and the common theme is how do I scale my nodes, right? Eventually that work uh you know ends up being delegated to a service provider like the ones that we know and love.

So I went on you know trying to explore this a bit deeper. And so some of those challenges were uh if if you have um a lot of a lot of requests uh coming up overnight if you're scaling you're a D5 player that h that happens a lot right you scale overnight a lot of pressure on the RPC right if you're running a lot of u predictions e calls and stuff like that. So horizontal scaling is what people revert to the most, right? That means if I have a you know a lot of demand for those ETH calls or those RPC calls in general, let me just add more and more

blockchain nodes of the same type to handle the uh peak of uh of access. But that creates three problems, right? One of those is the reaction time. So how the time that it takes from your access pattern to increase your demand to increase they need to have fully ready uh RPC node able to handle all those requests. So that includes a lot of technical work managing snapshots loading chain data the sync time and consistency checks health checks and all that.

Second forecasting. If you don't want to really overprovision on uh scaling u a lot, you have to be really really good at forecasting your access patterns. So then you scale intelligently and don't spend more than you should, right? And the final boss of that which is a big problem as well. I would I would call it the routing consistency. If folks are trying to access nodes, there is, you know, quite a few challenges if you need to route them to the same server always all the time or if you have a global consistency problem because each node syncs independently by design.

Welcome. Uh so you know if if we dig a bit deeper and try to see what what is the actual use cases folks are using uh RPC for um and you do a breakdown of them you can kind of come up with with a model there's certainly a pattern here you have some some methods that would always kind of return the same thing this static no parame no parameters so per the per a specific a given chain it well always be like that block numbers. Well, it is kind of a function of a single parameter anyways, right? Um you you can also have things like got logs, feed history, eth call. Those are

kind of different things that happen um um on on the back, right? So here with feed history is basically a computation of the the latest blocks, right? and ETH call is you know a bit more involved. I've seen a lot of folks in the space um trying to scale. They try to just come and cache a lot of these things here. this kind of three first and although they can, you know, really effectively catch that. It's it's not it's it's it's not really hard, but then you have, you know, some some sorts of problems here because ETH call is really the the big boss, big enemy,

right? uh because this here um is basically for every uh token balance call you're running each call and how many times you're going to query your you know uni balance of the block 5 million most likely only once right and then when when it comes to the next block then you have a different call here so the cache m hit ratio was really unbalanced here for eth call just to give a number Um, one of the customers they they they try to cache you know as as many methods as they could they could only have a cash hit and miss ratio of 17%. That's suboptimal, right? And we know that ETH

call is really uh more demanding and involves you know heavy computation side. All right. So we can definitely try to cache a lot of things but if you want to tackle a problem why not jump into the hardest part first? That's kind of a more personalitywise that's kind of a bit of me. But why don't we try to really externalize uh the state that is you know kind of a interesting idea because you can just put a database and then you can have your virtual machine calling making those as load calls but then you might run into some some other issues.

Main issue is with the latency of the reads right. So is and then I was kind of exploring to see if is there an opportunity to actually have execution and storage on a highly scalable you know database type with minimal latency for for s loads and turns out that the answer was uh was there always. So we basically uh I was basically exploring to use radis for uh to store the state for each of the contracts and I would have a first load of the data into radius uh then I I'll get into more details a bit later uh but having

both the contract code storage balance uh nons uh that is all you need plus the some of the block metadata That's actually all you need to execute an ETH call, right? So that's what I did. So users can, you know, still call G. Uh but they can also call um Radis for their EVM interpretation side. And but you know, how do we actually make this work? So, Radis has this extensibility um feature that allows you to embed Lua which is a interpreted language arguably one of the fastest one we have today with performance that approaches to to

see right and you know game on I had to implement a bunch of B codes and do the first test you know on the first initial test it was like you know more or less 40% faster than Gath even without loading um any data from smart contracts, right? And so that's what I did, right? So I had to copy um a lot of the data from contracts without the storage keys uh into RDIS. And here's one interesting thing. If you have, let's say, a a D5 protocol, you actually don't don't need the full blockchain data to make your calls. You can only have the ecosystem of your own

contracts loaded to this external interface that is plugged to your own uh running node. Uh if you trust the setup enough, right, that that can allow you to really uh scale horizontally with RDIS in a much faster pace. There's certainly a lot of innovation in the space these days, right? with Wrath, with uh Bizu, they have this lean uh node right now. Uh but I think this is still a valid approach because it is it allows you to actually load what you're going to need much more like a a cache uh mentality, right? You want to load into RA into the cache environment what

I actually need, then you start using it. So here's the project. It's called today it's called EVM Lua. You know I did a technical validation it actually you know it works. Um it is a micro EVM interpreted implemented in this language called Lua. Uh was created in in Brazil actually my country right and Leonardo's as well. uh it it executes inside radis making really minimal right the the latence uh the latency for for reads and well it is actually able to interpret ETH calls not only that but if you're estimating

the gas uh for a transaction that's actually also an eth call model it's just a return type that is different same with debug trace call right and so many other methods that are actually around just running EVM code. So, ETH call I picked this one because it was really the the biggest problem, the biggest villain over there, but it can be easily kind of extended to other use cases and EVM intensive calls as well. Now I'm actually very uh yeah so just for the deep technical minds here the prerequisites for execution node uh if you want to run it outside of g you need to have debug trace block method need to

have a pre-state tracer you know and and I do this using diff mode true which gives me for each block I got one call to the RPC and then I list all of the storage keys that changed from all the contracts that had changes I can selectively load into my database what I want what I need and you know call it a day and for only one call per block so that is you know very interesting I've seen this done uh differently so tracking state changes in different ways and I think that really keeping it very simple here um it would be key right of course that for a full production implementation

you would also need uh block metadata, right? Because from from the from the EVM uh you can also um have you know access the uh the block hash and and you know a lot of other block uh metadata and blockchain context there. So I believe that we are you know something in between but I'll write it here because we have uh over 100 uh up codes implemented but you know there are some key uh things here like add the proper metering integrate uh the official EF tests uh into it. So I'm actually you know testing on the

optimistic side today right so integrating with the real tests is you know it is the the the challenge here and you know the deployment phase it would involve the actual you know full-fledged benchmarks the optimization uh feedback loop which is man you just uh put it to run and understand the hot paths and you try to optimize that each of these segments And most importantly because it all ties back to the user try to see what are the use case specific features that are needed there. Why folks would actually use it for is it to to actually simulate transactions. It could be done using it

but it might require you know some steering on on how this is implemented. So this is not the benchmark of that project but this is the upper bar of radius right. So this is a benchmark that was done on AWS using the service that is basically managed uh radis elastic cache and folks were able to run with 1.2 million requests per second. Those were actually very simple calls. Now we're talking about EVM calls but even so that kind of gives us some sense of scale here and if you that is a single node right and that is a single uh and that is a

cluster right a cluster of that node with the replication on the database would handle all the replication for us in this case it can actually surpass 500 mill RP RPS so that kind of brings me to a philosophical moment here of hey if if you don't actually need to have the elasticity of scaling horizontally within your company within your project would that make it easier for folks to manage one or two Ethereum nodes right uh can we kind of you know is what are what is the opportunity here for the individuals that are using Ethereum today right or EVM in general so that's

kind of the thesis of it and I think it's really important that us at this stage where the whole blockchain thing is validated right is for us to zoom out and see what are the opportunities for us to integrate to other long-standing platforms and I think that was a you know good exercise if you folks you know want to give a follow want to see what's going on with the project um we'll be posting updates there that's the GitHub project event and thank you We can take questions if time allows.

Um when you say the you wrote like a micro interpreter in Lu what's the micro part is like a selected subset of up codes or Yeah. Uh thanks for the question. Uh it is around 112 117 up codes out of the 150. Yeah, that's the micro part. Yeah, but um I have to tell you that I that not the most daunting ones are implemented. So that's to get to that number wasn't particularly hard, right? There's a lot of pushes. I love that. Yeah. Yeah.

Exactly. Yeah. So arithmetic there are still some challenges there because um um uh the Lua version that's loaded into Radis is not fully capable and then even the arithmetic can be tricky, right? So um you know implementing EVM is is not a trivial job right uh you got to really lean into the the edge cases the happy path is really happy it's really fulfilling uh but the edge cases are everywhere right it's a landmine thank you right uh so just wondering if you if it could be supported to do like sub calls so static delegate calls these kind of

That's right. Storage. Yeah. So that is a very good that's that's a very good uh question because um it it is it will be possible to do these static calls and to do the nested calls. I think it is uh nowadays it's paramount right for you to implement something like that. Um because most of the contracts the relevant ones are under proxies and you know that's that's a needed one. Uh I think that in the fullness of time there might be uh even a feature I was thinking about is hey you load uh an ETH call uh code into a a script and then it would actually understand what are all the contracts that that ETH call

typically uses or it can monitor that using your main infrastructure and then it can load those contracts on demand into the the radius side and if the radis doesn't have access to the storage of that contract, it would just kill it and have GU serving as a back off as a as a backup. Uh so definitely I think that's a clever um question. Thank you. Um I know that some chains they do try to do outsource this job to some custom indexes for example Solan or EPAS and then saying hey it's up to the to the client to configure the indexer in the

way they want to access all the indexes working it's basically indexing all the effect the all the data that this particular application needs and then you just using graph kale to just ret index. What is the difference of this approach to what you described? Very good question. So, uh there there's definitely um increasing uh demand for specific access patterns for blockchain, right? So, that's a great question. Uh and I think that this here would actually shine on the DeFi space because you need to have you know real time applications.

Sometimes those data indexers, you know, it might be um there might be some challenges to catch up. It is very sophisticated infra to make uh you know Solana indexer work live right uh if we're talking here about you know hundreds of milliseconds of block time right so it might be very challenging for for real-time data apps uh to catch up with with with the index uh and I think well uh there's a difference between accessing state directly and using ETH calls that's the EVM interpretation right so you really um simulate um how would a unis swap trade would look like under a variety of uh trade paths and you could do this uh with

theoretically lower latency right so we'll get there thanks for the question very insightful hi um two questions one how do you deal with when your nodes don't end up on the canonical path because it does happen the more you run the more it happens you end up with some data that you index especially if you try to stay at latest block do you have like methods to to deal with that and the other side was did you ever think of just altering the way uh goeth stores data and taking that path rather than re-indexing that's right uh two great questions so um

I don't have full data schema implemented that would allow a revert be you know quick enough to to support it. Uh so what I have today is uh you know again the optimistic data loading. Um but I'm thinking about uh using the the diffs that I take for for each block into actually reverting that and you know re rebuilding it. Right? So that would be kind of a kind of a step letter if you're let's say three blocks behind to kind of try to revert it because the pre-states tracer actually shows me what was what were the values before that's incredibly valuable. So whoever designed that pre-state you know that that was a

good job. So that allows it to re reverse what was once truth right. Okay. And you'd use some maybe some consensus between your own nodes running to decide on the canonical block or something like that. uh I don't today but I think it would be it would be reasonable to do so u as the criticality of the use case permits right and uh to your second question would that make sense to have it embedded into a proper you know client I think it would right I think it would I think um innovation happens like this to try you know to try to prove something that it works outside of the main environment and then you know folks come and incorporate I have no personal love about the project to the point that

hey uh I'll try to have it own it myself. No, if anybody wants to take it that's you know it's it's open source for a reason. Thank you. Thank you. Yeah. Next question. All right. So we're out of questions. We're out of time. I think it's a good match. Okay. Thank you very much. Have a good day. Byebye. Thank you. Thank you very