Ethereum Layer Two Client Similarity: Geth

Transcript

Go, we can go. We can start. All right. Hey, everyone. Happy to be here. My name is Jan. I'm a co-founder at Zcash. Zcash does a lot of things. We had a layer two, still do. We have a Zcash finance product. We do a lot of lot of cool stuff. One of the things that I'm really happy about doing at Zcash sometimes is some research, some academic research. Some of this is cuz I want to answer my own questions. Some of them is because I want to convince other people of interesting facts. This is a little bit of both. And this is actually a talk about the use of Geth, Go Ethereum, in layer two clients because Geth is sort of everywhere. You hear this.

But do you have the the data to back it up? And if it is everywhere, what does it mean to be everywhere? So, I'm going to present this work. And this was basically a study to just do exactly what I said. Try to figure out if you're using an L2 and if it's Geth-based, [clears throat] what does that mean specifically? Geth is a very complex piece of code. There's a lot of things going on in there. I'm by no means an expert. I can, you know, touch parts of it, but I don't know all of it. Uh but when we start to see Ethereum grow beyond just the Ethereum L1 network, people want to do things like layer twos that are critically EVM compatible or EVM equivalent, compatible or equivalent, whichever.

And often how they do that is by saying, "Well, I'll take the EVM code base, often found in Geth. I'll fork it. I'll make it work in an L2, and everything will just work." And that is more or less true. It's very successful. I mean, it's I'm not here going to say that that's a bad approach by any means. But what that means is you have this weird net of software dependencies that goes on under the hood. And I spent some time doing software engineering research, so I wanted to know if this is sustainable or good or can we improve on this? And if not any of that, at least what What going on? What is what is the state of the art? And so we were motivated to look at L2 adoption.

Also a bunch of other chains were claiming to do this, unsurprisingly. And so in late last year we started to look at all the L2s on L2Beat. There's a lot of them. It's about 131 at the time, maybe 128 according to the slide. A lot. There's 130 L2s. Some of them are L3s. I won't make the distinction. I also won't distinguish between rollups or previdiums or various similar terms. Basically if they're a network that uses Geth and L2 L2Beat track them, we considered them. And what we wanted to see is how often Geth was there and then what was going on with those Geth codebases. And these were These sound like very easy questions to answer, but getting detailed answers actually is actually

beyond the scope of the paper. We didn't get enough done. But if you are interested, we did write up the results. It was a paper. It was presented at IWBOSE, which is a workshop at Saynor, which happened just last month in Cyprus. And I'm happy to send you the preprint if you send me an email later to talk about it. But to go on a little bit, what we found is 70% of projects in L2Beat use some version of Geth or claim to or are expected to, depending on you know how they are built. Tracking them was actually a little bit harder. And this is important because it's $55 billion that these systems collectively managed or at least that was the case in October of last year. And we found 31 open repos. And so now we want to look at these repos and the

state of the ecosystem and say, you know, are they updating in lockstep with Geth or not? And is there a good reason why they are or are not? So, let's get into it if the slide works. So, as I already mentioned, EVM compatibility is one big concern why you want to do this. If you're using Geth for its execution environment, you would expect that you don't change Geth within your EVM or sorry, within your L2. Possibly that is the case. Some L2s do make modifications. We'll get into that. But also when you do that, you do get these software dependencies as I mentioned. You start to see Geth being updated, and then other networks either falling in line with that or not, and that introduces issues with things like security. While we can trust fraud proof

since they're knowledge proof to ensure the EVM itself didn't sort of violate the rules you wanted, it doesn't mean there aren't issues that can still be present in those systems. If Geth pushes a patch to do efficient memory pool management, for example, and you don't adopt that patch, maybe you're subject to some sort of an attack that uh a well-maintained piece of software would normally avoid. And so we want to ask these questions, or maybe you have different peer requirements, or you know, maybe leak uh secret uh secrets are going to be leaked because, you know, you're signing blocks in a different way. Maybe that doesn't matter as much as an L on an L2, but also maybe it does. If we do see a lot of this, maybe, you know, it's time to go build a client that is actually having L2s in mind. Um certainly, it's possible, you

know, Geth could cooperate very closely. They might already do it. I'm not saying they don't. Uh but, you know, we wanted to see what was the state of the year. And so why now? Well, because 128 seems like a lot, right? When there were three, uh it was pretty easy to go track, see what's going on. Some of them have very good websites describing what they do to Geth. Optimism has a diff that they publish, or at least they used to. I'm sure it's still there. Uh you know, it was very easy to see what they what they what they changed. For other L2s, it's not necessarily the case. Um actually, what we found is a lot of L2s are are developing in the dark. This is a little surprising because we're as an ecosystem trying to say we're open, and we want to be

transparent in all of this, but only 31 repos on uh public repos for Geth out of 121 128 rollups. That's that's not a lot, right? That's a lot of hidden code. And so you can infer a lot of what's going on, but you don't actually know. Obviously, even if you publish the code, you don't necessarily know what they're running. There's some questions like this, but uh we wanted to see what's out there, like I said. So specifically, we had five research questions in mind. Uh Now first one was how accessible is this L2 client code? So if there if you're using an L2, do you know what execution client they're claiming to use?

Two, um Geth specifically, how common is it? We chose Geth because we anticipated it it would be most popular. The results did confirm that. We would love to repeat the study for Reth or other clients. Three, if you do have um Geth dependencies, what do they look like? Are they just all direct forks of Geth or are you going through a middleman like OP Geth? Four, uh what differences are made? You know, why are people changing this codebase? It's such a battle-tested piece of code. That's one of the benefits of using it. Why are you touching it? You could break something. Maybe there are good reasons, maybe there are less good reasons.

And five, uh if you see security patches or updates go to Geth, how quickly do you expect to see the rest of the ecosystem catch up? And is that important? That was actually the toughest one to track, so I don't have a lot of results for that, but I will try to get into everything. What we did uh was we looked through L2Beat to try to find all the public repos we could. The corpus construction was pretty straightforward. Um probably a little naive. I'm sure we could do a little bit better if we had more time. Uh but we wanted to go look at at repos. L2Beat does sometimes link to them. In other cases, we had to Google them. And then we queried RPC endpoints to see if they were actually telling us they were using Geth. Uh not always the case.

Optimism, for example, um sometimes reports Reth. They alternate. Uh and so as long as there was sufficient number of calls that returned Geth, we did consider them as part of the study, especially if we could find a Geth codebase. And then we found there were three ways people could fork Geth. Not surprising. First, you could just hit the fork button on GitHub. That was the easiest. You had a lot of upstream data when you did that. Second, you could clone it and own it. So you could download it, keep the history, put it into your own repo, keep the readme probably saying it's Geth-based. And then three was history-less copy. So literally copy the code, delete all the metadata so say goodbye to all the commit history and all of that, and then say it's your own your own copy of Geth. That was sort

of the most problematic cuz we really don't know um what's going on there unless we start to look at all the code. And given the size of the code base, it was very difficult to do this. Especially in such a short period of time. So, what we found, um 128 projects in L2Beat in October, uh about 89 of them were using Geth. So, actually Geth is pretty dominant. This is sort of matched the expectation that we had. Um Geth seems to be the main execution layer for L2s. So, that's pretty cool. But, of these 89, very few of them had public open-source repos. Um what that means is how they're using Geth is not necessarily clear, and they might be intentionally hiding it because of security concerns. Maybe they don't want people to look at their code. Maybe they're scared. Uh they don't want to

pay the bug bounties. I don't know. Uh maybe they're using a roll-up as a service provider. In some cases, these don't actually provide the open-source uh repos. Uh maybe it's another reason. Uh in some small handful of cases, we did find that they used to have uh a repo, and then it became an archive, either officially by being marked as an archive, or unofficially by just stop getting updates. But, if the roll-up's active today, and the last commit's a year and a half ago, I would bet that version of Geth is not the version that they're actually using in production. So, you know, that raises some questions about why they're doing this and what really changed in between.

Even then, as we'll talk about, uh the versions that are reported by RPCs are not necessarily um foolproof, right? You can cheat. You can change that version code. In terms of accessibility and and prevalence, so that means that um public execution client code is often missing, despite all of this. So, you again, you might have systems that ensure that the state update is the one that you want, so that if you download the code and you run a fraud proof or a zero-knowledge proof, things do work, but it doesn't mean that the system is as as robust as it could be. In other cases, uh and we often found that there was um you know, repos that existed, but didn't necessarily match the version of Geth.

It would be an old repo with a new modern version of Geth, so something was definitely not going on. So, this is just to say people aren't necessarily as transparent in this field as a lot of the marketing teams would say. And admittedly, ZK could to you know, publish first uh private version and then a public version. So, we can see why we do this sometimes, security for example, but uh it this is just the the way it is, it seems. So, now if you someone says, "Oh, yeah, we're all open source." It's not necessarily the case. A lot of the the lineage is very varied though. Geth has been around for as long as Ethereum, I think. Uh certainly it seems that way. And there's a lot of ways people use uh Geth in their L2s, which we found very interesting. This is a graph that uh sort of pulls out a lot

of them. Um there are 128 rollups missing from this bottom row, essentially. But um what we actually see is that a lot of the um Geth-based rollups use an intermediary version of Geth, one that's been sort of modified for layer two um execution. Very popular with Nitro and OP Geth, but there are some others. And then even when you use OP Geth or Nitro, you're not necessarily doing it with that specific version um because L2Beat might say that you're using a rollup as a service provider, in which case they might be doing other modifications. Now, I'm not aware of them actually doing other modifications, so it's not necessarily a bad thing, but it's uh not easy to confirm, and so we just have questions. And we don't know what's going on there. Doesn't mean you have to be concerned, it's just what it is.

When we look at the RBC data to confirm that the uh graph, uh most of them do come from OP um Geth or Nitro, so these are by far the two biggest rollup stacks or frameworks that you can use to build rollups or L2s. Uh and OP Geth is just a little bit in the lead over Nitro. Doesn't matter, it just goes to say that if um you see OP stack chains being updated regularly, that means you're probably affecting a larger set of the the the um ecosystem than others, and similarly with Nitro. So, we want those two particular teams to definitely be keeping up to date, and I'm not saying they are not. Like I think they're doing a great job, um but certainly this this is the type of dependency that we wanted to identify because if if people are saying they're OP stack, but actually most of them are

doing their own custom stuff, or they're using only the OP node with their own version of OP Geth, um which is not the right. That would also have been a very interesting finding. Uh okay. And so, what happens when you actually do start to change some of this code base? Geth is complicated, as I said. Why would you want to change it? Why would you want to introduce breaking changes or risk breaking changes? One is because sometimes you have to. You have to introduce L2 specific stuff, whether that's block building, ZK does stuff with transaction including this bunch of things. But we also found things that were pretty common like system transactions. So, you want to get the state of L1 onto the L2, but you don't want to pay for it. That means maybe you have a different transaction type that you wouldn't normally find. That will

obviously have effects for your validity system at the end of the day if you go with that. But this is something that an L1 would never consider because you'd always pay for gas on an L1. On an L2, that's not the case. In other situations like Taiko, Geth is used in very creative ways. They have it to produce to collect transactions as a memory pool and then later as an execution client. So, they have it in sort of two different modes. Um you know, it's something that's unique to based rollups and as we see more and more based rollups, we may see more similar designs or maybe the Taiko code base will become a standard. I don't know.

That's just speculation. Um three, sometimes you do want to change EVM compatibility. Maybe you do want to change how things are executed. Maybe you want to introduce opcodes. Boba, at least in the past, did have a situation where you could have an opcode to call for data to be read on a web two source. It's pretty cool. Does require some changes. Again, not something that necessarily you want in an L1, but in an L2, you can play with all these rules, especially, you know, the recent conversations about whether or not L2 should something scale things. We can change things on an L2 and I think that's totally fine. And this is some ways they did it. And then in other cases, what we found was either consensus or how these nodes talk to each other. Some people wanted to change it, whether they just wanted whitelisted actors or you

know, some do something different. And often we saw teams trying to optimize Geth, trying to to to you know, get caches to read faster or change sizes of certain data structures um for for production reasons. Um some of those can probably be pushed upstream, which is actually something we'd be we'd be very interested in seeing because if, you know, you're an L2 and you optimize a whole bunch of features in Geth and it doesn't break Geth, maybe Geth wants that, too. Uh but anyways, these are some of the reasons we did it. Uh to actually understand all the reasons why you would change Geth in an L2, you probably have to look at 128 code bases almost, you know, uniquely because most of them do something a little bit different in some way or at least they've just evolved in a different way naturally due to

software engineering processes. So, that's a really hard question to ask, but we did at least get a favor. If we continue the work, we may dive deeper and be more comprehensive. On the final case, uh we want to see update behavior. This is really hard track. The ideal situation would be to look at the specific set of security patches that we were concerned about on Geth and then find the code bases for the specific L2s and see if they'd implement the right change. This is not always clear to do. Again, 31 rollups uh or networks were the only ones that had open source um repos. So, for 50 of them, we're not going to be able to do that. What we can do for all of them is query the RPC and say, "Hey, what version are you running?" Now, you can lie about that, you can change the code,

uh but actually we found that most of them were fairly recent in a lot of cases. And if you look at the two largest family of Geth dependencies on layer twos, uh for Nitro and OP Geth, for example, most of them are in pretty recent versions. So, most of these are being updated. We don't have any reason um to suspect, you know, falling behind significantly as a problem, uh but obviously there's generally a reason people push new versions of code. So, ideally they all would have been up to date, but we do understand that actually running these things from an operational side of side is is non-trivial. And so, you know, just saying update the node code rarely is easy as it sounds because you have partners, people running your nodes, and specific things that you may have to adjust depending on patches. So, to have it actually in the the

it's pretty good. The other thing we wanted to track um to see this is go go-ethereum, right? That's what Geth is. Um are people using the right version of go? Um we're not aware of any situation where the language itself could introduce a breaking change into the client. Uh but it could happen. There have been cases in the past where a language has had a a problem and you need to update your version of that language to overcome that issue. In this case, we did find that over 70% were up to date with go, so that doesn't seem to be a real problem. Go itself is pretty, you know, mature and well-maintained. I don't anticipate this what to be a problem, but we wanted to see if this would be uh something of interest.

And that's about it. So, in short, we want to look at how Geth was being used in L2s because there are implications for this, right? First is um patch propagation. If you have vulnerabilities in Geth, which may apply to an L2 setting or someone running your piece that piece of code in a cloud somewhere, we want to see if it propagates. It's hard to track this. We have to continue doing work on this if we're if we're interested, to be honest. Um second, we wanted to see, you know, if if these patches would mean there's correlated risk if we see a problem with um one L2 and then we see another L2 having a similar version of Geth, are they going to have the same problem? It's something we could track. We don't have any examples of this happening, which is great. I don't want any examples of

this. We want to make sure we're ahead of this and paying attention. Uh third, we really did find that not a lot of chains are being as transparent as I think the ecosystem would like. All right, so it's very surprising that um still in this day and age, most people don't bother to sort of push for relatively maintained, if not perfectly up-to-date, um versions of Geth. So, I hope there's good reasons for that. I hope they're doing that for security reasons and not just laziness, but I think it'd be something that's really nice. And um third, you know, there's a potential for semantic uncertainty. This seems very less likely if you do have a working fraud proof or validity proof system, but if you're especially not a stage one or two, whichever appropriate stage roll up, and you don't open source your code, and

you don't like have other people validating your state roots, maybe this is a concern, cuz maybe you broke something, or maybe you did something by design that's different that you didn't tell your users. So, um that's about it. I will say the study's not perfect. I tried to say that already throughout this. I think it was a good start. We'd love to continue to working on it, or look at other clients. There are some limitations. Um first, again, if things were invisible, we couldn't do very much about it. We didn't go knocking on doors and say, "Hey, give me your code." I don't think that would have worked. It would have taken a lot of effort even if it did. Second, we looked at get diffs and lines of code change. Doesn't necessarily paint a good picture. You can implement the same like a security patch possibly

in different ways. So, it's not necessarily the case that lines of code change matter. The paper will have those numbers though. Go take a look. Unfortunately, I don't you know, have the slide time to do it. Um third, RPC versions can be lied about. Already mentioned that a couple times. And fourth, public repos may have real good reasons for doing this, which is fine. But then, I would like to understand what those are, and people should say it, not hide it. I think that's that's a fair ask from the community. In the future, if we could look at other clients, that'd be huge. Um Reth is gaining a lot of popularity. There are definitely other clients out there. Um second, if we could find out how the patches are being applied um semantically or using static analysis, that'd be really really nice. Lines of code, like I just said, do not paint a

good picture. There are tools like Blockscope, which is another research paper that say, "If I have a security patch on a firm and another Ethereum client, did they apply the patch?" The problem is you need a good list of security patches on on the original Ethereum client, like Geth, and I didn't have that when we started the study. So, we'll have to do it later. Um they also do it do that for other L1s. So, this is still complimentary to what we did. Um what would also be great is if we come together as a community and standardize patch processes. Probably for the biggest players in the space, this has been done. But as someone who operated one of the sort of smaller roll ups in the space, you know, I was not aware of any of these and it would be great if we could, you know, build out

that process so that it's a little bit easier to keep everything up-to-date. And finally, um, we can also monitor updates in other ways, right? When Denker and Petra hit, for example, we could see different behavior in, um, the blocks that are being produced and so we know versions are being updated. Do we need to do that because people are not telling us everything? Or should we do that? That's another question. So, the takeaways, yeah, Geth is everywhere. I'm out of time, um, but essentially Geth is popular in L2s. We should start acting like it's, you know, a relevant critical software dependency for L2s and we should, you know, treat it all as a as one good ecosystem. So, I'm out of time.

There should be a thank you, but I guess not. But, thank you. There it is. Thanks a lot. Um, we can take one question if you have any. If not, then I invite you to continue the discussion outside then. Thank you. Thanks a lot.