The Hidden Costs of Running Ethereum on VMs

Transcript

Hello, hello, testing. Can you guys hear? Okay, all right, let's get started. So, hi everyone. My name is Victor. I'm from the Latitud team and this is actually my third ETHCC, but the the first one I'm presenting. So, for me it's a pleasure to to be here and uh I chose a topic that it was quite common during the first three editions that I was here starting in Brussels, which is the the hidden costs of running Ethereum on on VMs. And the thing is like ETHCC is a very welcoming space and we have like everyone coming by. People who know a lot about Web3 and people who are just getting started. So, just a quick disclaimer, when I'm talking about VMs during this whole talk, I'm talking about the virtual servers that we rent in cloud computing companies to run the

the workloads. I'm not talking about the EVM itself. You okay? With that said, let's get right into it and the best way to start is to just go back to where Ethereum how Ethereum came across, you know? Back when it it was designed, its goal was to everyone to be able to easily take part into it, you know? So, coding should feel just like the way coding for for web development felt like. And the infra requirements that that we had in the beginning it should be like people could run these things in any device they already on, you know?

And that's exactly what happened, you know? Solidity just borrowed the syntax from JavaScript, so writing smart contracts was quite easy. And the requirements back then was just anything that we could find, you know, in stores for for consumer grade. And yeah, that was amazing because that's what set up Ethereum for its success, you know? And the issue was as [snorts] Ethereum started to grow and reach new heights, so did its requirements. So, in order to succeed in the protocol, these co- companies, they had to reinvent, you know? So, when we talk about validators, RPC providers, even roll-ups, they had to reach consumer-grade hardware and go to the public cloud. And the thing is, it was

important because all the performance improvements, they were noticeable. Some of these companies, they were even not cloud-native. They didn't even try, they didn't even bother to run a consumer-grade hardware, you know? And in order to better explain why there is, I think it's good to do a an analogy with web two. Uh so, I borrowed this diagram here from Bert Hubs' uh newsletter. I'm not sure how much you guys are into cloud computing, but I totally recommend you subscribe. He has a version in Dutch and a version in English. And this is how a full-stack solution it looks like, you know? And this is how most companies they operate. They ship software, you know? So, the way they set up internally is they have teams taking uh care of the upper part and they outsource the bottom

part because why should they, you know, bother with that? It's not part of their core business, you know? And yes, it does it did make sense like 15, 10 years ago, you know? Because companies like AWS, and AWS is just a representation, you know? This could easily be Google with GCP, Microsoft with Azure, or even Oracle with OCI. The moment that they start to make a lot of success with cloud computing, they start to offer other services, databases, storage, private connections, managed communities. And then what looked like this in the beginning gradually started to look like that, you know? Companies they were running, I don't know, maybe 90% of their stack in a single provider. Uh that's an issue, you know? And these are numbers from Q3

last year. Basically, 2/3 of all the expenditure in cloud computing is centralized in just four players worldwide. This is web two and web three combined. So, at the end of the day, decentralization just lost priority to scalability and flexibility. And the thing is, if you look what there was before the public cloud, it makes sense, you know? Back then, if companies they wanted to run something, they had to procure everything themselves. They had to reach out to a server manufacturer, they had to go over the specifications, wait for the the order to be manufactured, delivered to their place.

Once they they received that in their place, they had to procure electricity, network connections, uh refrigerated space, and make sure it was all redundant, you know, to have uptime. But, when the public cloud came together, all the companies they could scale up and down as easily as pushing a button, you know? And they were paying only for the allocated compute capacity. They were not paying for everything. And as you guys could see in the previous slide, procuring complementary services was just super easy, you know? So, before the public cloud, there was like a huge investment for you to even get started on coding.

After the public cloud, was just higher opex, you know? You're paying more for services, and you can focus on your your core business. And this is what it looks like, you know? These cloud providers, they are the ones purchasing the hardware for you. And and this is how they stack everything. The hardware underneath it, you'll have a OS that they put together, and then you have that very important layer called the hypervisor. The hypervisor is what's going to be partitioning the that server into a smaller VMs. And this is what we rent when you're using their services.

That small portion over there that we also install our own OS, and in case of Ethereum, all the clients on top of that OS. You know, there are pros to it, of course. You have elasticity in your infrastructure, you're not spending that much money anymore. But, there are also cons to it because you have just a tiny fraction of the underlying hardware, there's a huge markup when you centralize your your compute only in four providers, and there are also locking issues when they're running so many things with them. So, is it really the ideal solution for Web3?

Long story short, it is not. That's exactly what we're going to be seeing here. So, when we talk about general workloads running in the public cloud, there are two main concerns that we have, and that we see their impact. The first of which is performance, you know? Uh you're sharing everything underneath the server. Disk I/O, CPU time. You're also more susceptible to network jitter because although the hypervisor caps what the VM is consuming, all those smaller VMs, they are running on top of a server, and that server has a very limited number of ports. So, we are also susceptible to network jitter. Resource contention. And that's just one side of the story. The other one are the costs that go beyond just the computational

costs because as you scale up, you start paying more for bandwidth, and you start paying more for storage as well. So, you lose sight of all the predictability that you have as a company relying on them. So, how does that impact Ethereum specifically, you know? I have a few examples here, like the execution clients. This is one of the most impacting clients uh from the VMs because they need consistency when they're reading the the hard drives, you know? And the variability that comes on running on top of VMs greatly impacts that. The same is also true for the consensus clients because although they don't have as many resource requirements as the execution clients do, uh for them latency is quite important.

And what I what I mean by latency is not only have the servers close to where the transactions are being validated. All those layers that you saw over there, processing through all that software until it gets to the client takes time, and those are added milliseconds, you know? And there are even specialized workloads, you know, like validators who want to boost their maximum extractable value, RPC providers, roll-ups. All of them, they have a specific requirements that the variability from VMs is affecting them. So, I mean, at the end of the day, is there an alternative that to to the VMs? There is, and it's called bare metal.

Uh for those of you who are not familiar with uh the jargons, bare metal is what how we call fully dedicated servers. These are servers that all their components, all their resources belong to a single tenant. There are no neighbors consuming the resources from it. And one might think, but isn't that overkill? Why would I need a whole server for myself? The thing is, you don't need a whole big server for yourself. You can also rent for yourself smaller servers, four cores, eight cores, 12 cores. You can put together the specs that make the most sense for you. And how does it that impact node operators for Ethereum? The moment that you ditch the the hypervisor, you have direct access to the hardware. And with that, you can tune the server according

to your own needs. You're also get more cost efficiency because uh you're paying a flat bill for that server instead of paying for things that are scaling up and down. And you're also supporting decentralization because the moment that you ditch the four big providers and you start going for additional providers, you're reducing the concentration, and these guys, they're also not only operating outside of the data centers of the big guys, but also in the new cities and new regions, which is really good for the protocol. However, there are still some trade-offs you guys should have in mind because uh there's no one-click auto scaling. So, if your project starts to gain traction, you need to procure more capacity yourself, and you need to have people inside your company with enough

knowledge to set it up. But when you put both together, you can see that the upper layer is exactly the same when you're running on a VM or on bare metal. What changes is what's is underneath it. And the trade-offs, they come with a huge boost in performance and cost reduction. So, they are really, really worth it. Uh but, instead of just taking my word for it, I would like to just go over a few case studies that we have. This is actually a sneak peek of an upcoming one that is coming over the course of the next few weeks. Boundless, they have been working with us for the past months, and they have moved 100% of their ZK validators from AWS to Latitude's bare metal. And in doing so, they got 60% cost reduction on the cost

of those validators while getting twice the performance. So, it's not one instead of the other. They're reducing cost while boosting performance. And they're just one of many other companies that have pivoted from the VMs. These other guys here are also case studies that we have of companies that put together layer one infrastructure, RPC providers, or even validators of other protocols, and they ditch the VMs from from AWS, Azure, and Google, and moved to bare metal. And in doing so, they starting saving from 30 to 86% of compute cost, and sometimes even 100% of network cost. And all of that while having improvements from 1.5 all the way up to three times in their performance, you know? And

yeah, it all seems really, really nice and good, but how can we get started, you know? One thing that is super important, and this applies to all of them. They don't move everything to bare metal. They move what is most intensive, you know? The most intensive workloads are very expensive like uh workloads, they move that to bare metal. And they run the lightweight part of it and the bursty services on VMs. They still have other things that VMs are really good for. That's why they were invented in the first place, you know? And they keep doing that. They're running hybrid. So, when you put together like your own hybrid approach, start measuring.

Compare. Compare performance. Compare cost, and see. At the end of the day, you see that bare metal has a really good impact, you know? And well, uh before I wrap up here looking ahead of us. I'm not sure if you guys saw Jerome's opening presentation this Monday, but it was quite important because he mentioned how much it even has changed over the past 10 years and how much ETHCC has changed alongside it, you know, like you had you had to evolve. So, the conference evolved alongside the protocol. And the same thing is true for the infrastructure lead, you know, it is maturing.

When we look into the move from consumer grade hardware to the public cloud, it was the first move and it was necessary. And so is this next move, you know, so node operators, if they want to thrive from now on, they need to lower their costs, they need to improve their performance, and they need to become more resilient. Uh the good thing is that if they wanted to do that bare metal, that is already possible when they're running everything on top of bare metal. And that's it for today from my end, guys. Hopefully, it was as insightful for you as it was pleasant for me. You can check out all Latitude's resources, including our bare metal offerings, in this QR code here. Uh also, when you go there, you can also register for after

party that is happening later tonight in the Villa Branca Beach Club here in Cannes. And if you do have questions, if you do have any things uh specific things that you want to discuss, we also have a booth at the forest floor, and I'll be more than happy to welcome you guys there and go over your questions, okay? That's it. Yeah, a little bit. We're already happy. Anybody has a question?

I believe it's Yeah. Okay. Well, thank you, guys. Thank you.