The Hidden Layer What Providers See That Clients Don't - Bastien Moyroud - Alchemy

Transcript

All right. Can everyone hear me? Cool. All right. How's everyone doing? Good. Are you still excited about this this summit? I certainly was very excited. The only reason I came to Argentina was for this summit to be honest before I even knew I was going to give a talk. So, um, very excited to be here and congrats to Gary and Leandro and that was super interesting. So, thanks for running us through that. Um the talk I'm going to give is going to be a bit different in that I'm going to give you the perspective you know from alchemy. Oh so actually can we go uh I have the clicker so I'll go back one slide. Um we don't have the name of the slide. It is called TBD. It was going to go one or two ways. I was going to share how alchemy infra is built and how we

interact with nodes. I thought okay that's interesting. I can talk about it afterwards if you guys are interested. Um but instead I we have really cool insights running these binaries and clients at scale. um something you know the what I noticed the client teams don't always have visibility into. So I wanted to kind of give you guys insights into what we're seeing at Alchemy. Um cool. So that that'll be that'll be my talk. A bit about myself. I've been in this space for a while. I first started um in the Oracle space. So I spent multiple years bringing offchain data on chain. First with an extension to the TLS protocol uh and then uh being CTO a co-founder of a company uh I created afterwards. and I joined alchemy roughly uh I mean it's getting close to four years at this point. So um I've been

very involved with the engineering work we've done in Alchemy and also working with uh our largest teams and the way they interact with uh Alchemy as a provider. So um I'm happy to talk about all this afterwards as well. So I'm going to be here all day. So please come say hi. Cool. All right, let's get into it. So kind of four sections. I actually did the slides this morning. So I think I beat Mario to it. That's also why there was TBD on on on the slides. But um I wanted to break this down in mostly four parts. I think insights is probably going to be the most relevant for client teams here in terms of what are we seeing you know um since we started Alchemy. We've seen trillions of requests. Um so it's really cool uh data that we have uh there. And then also

wanted to highlight some of the custom work that we have done on some methods. I'll touch on this in a bit. And then also give you a sense of what we would love to see and work on with you guys, right? like from the the client team perspective and then future plans and where we can um continue giving you more insights. That's something we've started doing with some of the client teams, but I'd love to expand that to uh more teams as well. Cool. So, just some numbers. Um we're processing, you know, billions of requests a month since we started. I mentioned this trillions of requests total. Um we support 80 networks at this point. Frankly, I stopped counting. I think the number is actually much higher. Um, but we built infrastructure in such a way that we can extend rapidly to support more networks. And then I I

put here 20 plus binary clients if you count for the forks. I was trying to pull the number actually um talking to the team uh earlier this morning. But um I actually think it's more like over since I've been there certainly seen a range of clients and ranges of forks that we have given the amount of networks we scaled on. So and then in terms of where infre is deployed today, we we run these binaries in in four region and um we're also able to um on a per customer basis extend that out to other regions. So um we have data in these geographies as well. Cool. So in this in this insights bucket we have a ton and I'll also have a slide to show you the other insights we have and please come talk to me if you're interested in some additional numbers.

For this talk I wanted to scope it to two main buckets. kind of give you a sense of okay like what does the method distribution look like? Gary touched on the the linear airdrop and having a ton of ETH calls um direction that's also what we're seeing um and also give you a sense of like are these requests actually mostly close to the latest block or are they distributed um you know like archive requests or hitting blocks that are um let's say 1,000 10,000 100,000 blocks in the past. So hopefully this is helpful. Um and again if you have other other thoughts after this please talk to me and um I can share more data. Cool. So I'll start with this this this slide here. Um to start I I will say like when I was pulling these numbers up the top 10 methods in usage like account for 90 95%

of the overall usage and the the largest chunk is e calls. So to me it seems like most people probably have that intuition like if you're interacted with a node in terms of like what the distribution might look like. You might not see this at the scale that we're seeing it. So I'm you know hopefully this is useful as well. Um this is on Ethereum mainet though the distribution looks relatively similar on other networks. So these are really like this is a percentage. So really the it it actually when I pulled the number up I knew it was high. I didn't think it was that high like 53% is actually huge. So you know any work that we've seen on binaries that improve that performance obviously for end users that's a huge win. Um and then you obviously have your ETHA block by number ETH balance. I listed ETH chain ID here.

Um actually on the alchemy side we actually cached this and and serve it at a different layer but um you you get the idea. So yeah if you want to take a take a photo as well you're you're more than welcome to. Um, but that these are interesting numbers that um I I ultimately I think we will make them these more public, but uh for now I guess you're the first to see them. Cool. Um and then the second thing I want to touch on is uh I I alluded to this early on is the block distribution. So what are the most common access patterns that we see large teams use? the vast majority of requests uh today hit basically within um we we have that granularity to see which blocks are being accessed relative to the latest block that we're also aware of. So 80%

of requests actually um will be uh hitting let's say latest block minus 100 uh 100 bucks if that makes sense. And then we also have 20% so uh of calls past this 100 block. Now there are a few call ads which I actually should have listed here is I I showed you early on the this is heavily swayed by the ETH calls given they represent the vast majority of our calls right like 50% that drives up this number on on the left the 80% um because the ETH calls if you were to look at um ETH calls specifically 90% of the requests are within the last uh 100 100 bucks. So um you know people are reading data off the blockchain and they're looking to get the latest state. But if if you actually exclude ETH calls um 60% of requests are

within the uh 100 blocks of the latest block. So um you do see some variance there. So I I listed some examples here. Um mostly for teams that are looking to to back fill data from let's say the genesis block to the latest block um or you know debug or eth block receipts the numbers are actually very different. You know you only see roughly 40% of requests within 100 blocks. Um so yeah there's some variance there and again we have that guarantee and something to share. So um one one call I do do want to mention as well is we run a mix of full and archive nodes for multiple reasons. Historically we um were pretty keen on only running archive nodes for the operational complexity meaning it [clears throat] was easier for us to spin out archive nodes even if there was you know the syncing time and so on. Um

we've actually moved to running a mix of full and archive nodes recently for performance reasons. How am I doing on time? I know there's no >> Okay, cool. All right, so more data. Um, these are example of things that uh we can share with [clears throat] a caveat that we only um for some of these data points, we only store that data for seven days. Um, but we're able to, you know, share the method distribution. I call it the block distribution, though I think, you know, there's probably a better word for it. the latencies that we're seeing across various clients that we run, the error rates, be it the status codes, the um the node error codes, the RAM CPU usage that we see across multiple hardware types. Um I didn't touch on websockets here, but we

also see, you know, we have details on what the subscriptions are most used for. And then I really wanted to add a slide on logs. I think it's actually super interesting to understand like the query patterns, what people are actually doing for logs in terms of the typical block ranges, the most common topics and so on. Um I didn't have an opportunity to do that but we we do have some data. It's a bit tricky to pull but we're able to get that as well. And um I I mentioned this as well. We have per network per method per binary uh granularity. So we can slice and dice this uh data as needed. Cool. Okay. So the second thing I wanted to talk talk on uh talk about is the uh custom work that we've been doing. So by custom work I mean we've been in

situations at times where customers have asked us or teams have asked us to support a method um before the actual binaries might support them right so an example I listed here is alchemy get transaction receipt I'll touch on that in a second and then een roar transaction sync which is a more recent one um we have more examples but these are two I think uh really cool examples so um on alchemy [clears throat] get transaction receipts so I have the chart here but um I don't know how how familiar people are with it. But historically, if you wanted to fetch all the receipts in one call for a given block, you mostly would have to fetch the block itself, get the transactions, and then you know uh for each transaction call ETH get transaction receipt or you you could batch the calls, but it was still pretty um

painful for teams for teams to use. So in 2022, pretty early on, we launched our own method, so to speak, that basically did that under the hood. So it would, you know, do the batching. It would fetch the the transactions in a block. it would do the batching, call the receipts and return it to the end user. So that's this this um orange line that you see down here at the bottom. That's the alchemy specific method. And at the time there was actually you can't even see in the blue line, but there were some clients that supported ETH block receipts which was pretty pretty new and we saw 15 times more usage on our method than the binary one.

But the moment actually there was increased excitement in the space and more binary supporting it. I mean you can see this like skyrocket in terms of the usage. In fact, I've been recommending uh to teams to use the the native method, so to speak, not the alchemy specific one in part because teams want redundancy. They want fallbacks. They sometimes run their own nodes as a fullback. Um I'm very keen on having the binaries support these use cases. And I'm showing this more so to say I think we have a very good uh perspective working with a range of teams around what these teams might need. And going forward, I'd love to, you know, push forward more methods or share some of this data early on, right?

Like there's probably a delta between 2022 and 2024 here, two years from the moment ETH block receipts actually became popular and implemented across clients. But I think I mean look at this graph. Fantastic. Eglo receipt goes up and then we're basically going to eventually deprecate um alchemy get transaction receipt. All right. Um this is also an interesting one. Uh you might have heard of it. Uh if not, ETH get uh sorry, E10 roar transaction sync is a way for well maybe taking a step back. Historically if you are sending a transaction I need the water.

Thank you. I know I have five minutes left as well. So I do want to touch on this. Um cool. So historically, if you were to send a transaction and then get the receipt back, what you would do is you would use the send ro transaction, send a method, and then you would pull for a receipt. Now with the introduction of flash blocks um where blocks are sub 200 milliseconds, a lot of teams um don't necessy want to do this polling because the overhead that you incur polling for a receipt from a performance or latency standpoint will impact you. So, one thing that we did actually pretty early on when the spec was still very early, we work with Sam High and the the Rice team to build this in our in our routing or blockchain engine before the binaries um ended up supporting it. And now as

more teams are rolling out, I know GU rolled it out very recently. Um Reth also uh supported it. Um I think Aragon also has it too if I'm not mistaken. We've been effectively just routing the request directly to the underlying binary. But we do have this ability uh where it makes sense and to support the teams that use us to uh build some of that functionality in our layer uh in our blockchain agent to to start, right? Um my personal opinion is we prefer not to do that, right, for compatibility reasons. So in this case, we work super closely with them to make sure that whatever we were building was spec compliant until the binaries uh caught up.

Uh cool. I listed some other type of work that we did as well. We we pushed I mean this is huge kudos to the GU team as well like in been adding block time stamp to ETH get logs based on a ton of requests for the teams that were using us also work with the GU team to optimize the archive node performance on on base specifically um and an example [clears throat] of the data we had shared with the GE team again is for ETH get proof uh the team was looking to um figure out you know what was the block range that they should support or uh for how long for how many blocks they support ETH get proof um so we work with them on that as >> [music] >> Is this the end? Oh, no. Okay. Just a little uh ambiance here

wish list. All right, cool. This is uh I I'll be pretty brief. Um if I were to classify two buckets of things that I'd love for us to partner on with the client teams is standardizing error codes. I think one of the biggest blockers right now is client diversity [music] and having us running um clients. Having done this with ref where we worked with them to close 40 differences with guest is there are nuances um very often quite painful between uh the types of error codes or error messages returned from a client to the next. So um standardizing that for the community I think would be awesome and would greatly increase the client diversity. There's an initiative to do this. I know there's a PR like from a few months back um around this uh and

there's also the RPC standards call that happens bi-weekly on Mondays where that has been talked about but so far it's still pretty early on and then another piece is more so around tracing um that there are some nuances from a client to the next around you know the debug and trace name spaces beyond the differences between each namespace that is expected some clients might return um different data if you use the same method um within these tracing namespaces so I would say these broadly speaking are the two things that that we think like would be a to a huge benefit to the space if we were able to standardize this.

Um, cool. So, future plans on on our side. Oh, I changed the emoji as well. Interesting. Um, so, um, we're going to continue investing in best-in-class performance. From our standpoint, that means building really scalable, um, and reliable infra. We're going to continue investing in client diversity. Um, especially I on the execution side, like we we actually do a very good job already. I think on the consensus side we can certainly improve. And then we have two functionalities that we've been building over the past few months. One is benchmarks. Our process to benchmark nodes is fairly manual today where we built a system to automatically and um run load tests on various binary clients over various configurations of hardware.

So I'm also happy to discuss this and ideally the goal is we um we um we benchmark these clients um on our side and can share the data with uh with you guys if you're interested. Another thing we worked on is a more granular traffic shadowing where we can mirror real world traffic that we're seeing on Alchemy to a set of binaries. So um we're able to do that. And then the other thing that we have a few people are aware of it u because we don't really advertise it. We also have a um alchemy snapshots where we're starting to add more and more snapshots um to help the community. And then um I listed this last item as TBD.

When I say reports, I I'd love for us to as a company be more public about some of the insights that we have or or share more blogs around um the performance that we're seeing from various clients. You guys tell me if this would be useful, but uh that's something we're also thinking about. Um and I had two two small things I did want to highlight. A lot of uh people ask me, hey, like what clients do you run? uh you know what versions you're running and so on. We actually had added a change log or more detailed change log so to speak um three months ago. Um it's at alchemy.com/dog/change logs. So you can also have a look there.

And then the other thing I just touched on as well is we um at alchemy.com/snapshots we've started adding snapshots for uh multiple um clients. Cool. That's it. I did want to take one minute to thank you know Cena and the F team for organizing this. This is awesome. I hope this continues and it's awesome to see so many people so early on. So, thanks for running this. I mentioned the G Aragon Ref Lighthouse teams, but frankly there's so many other teams that we work with that I feel almost bad having to highlight a few. And everyone building the ecosystem. Thanks so much for the work you're doing.

[applause] [applause] Excellent. Bestie, I'll have you move to the center of the stage here. We do have a number of uh questions from the audience here and people have figured out the uh polling system in here to start uh ranking them here. Uh let me start with the uh first question here which is do you do you run your in infra yourself colo or do you use a cloud provider? >> Oh yeah that's an interesting one. We actually migrated from AWS roughly a year ago to uh fully bare metal. Um I mean I we have some really cool stories around this. um to keep it simple. It was both for performance and cost reasons and also we rebuilt our stack to easily deploy it to other regions if we

if we do need to. >> Do you use a single or multiple clients for mainet? >> Yeah, that's a good question. We tend so that's where the standard standardization come in play for every network. We tend to run two binaries but we route traffic usually to one one client. Um our decision there is mostly driven by performance because that's what the end users of Alchemy are looking um looking to take advantage of though we do have you know redundancy for uh client diversity. >> Uh besides the redundancy of the setup do you modify them in any way?

>> No. Uh I mean in the four years I've been at Alchemy there was um I think we had modified bore in the past and we had also like contributed to um our team had contributed to changes on at the level DB level. Um but I would say no directly we we try and avoid like having to modify the clients. >> Okay. And then based on the patterns you see do you have any change recommendations for the RPC structural or otherwise? >> I think so. It's interesting because I was showing the method distribution where ETH goals is directionally the the largest request by that I I interesting though like other methods that weren't even on that slides are equally important for teams. Um I you know I mean come talk to me. I'm happy to talk about some of the changes I would suggest. I I I think it's it's pretty

nuanced and depends on on what we're talking about specifically. >> Okay. Can you share the ratio of a full node to archive nodes? >> Yeah. Uh actually I don't have that top of mind. Historically it was fully archived nodes pretty much we had very few full nodes. Um but we did experience issues uh as there's been more network activity specifically on base that basically got us back to run full nodes. I I mentioned this early on as well. Um the idea was to simplify our deployment of nodes and we did a ton of work as well to sync nodes in a very efficient and quick manner but um I would say it's still pretty archive node like dominant I I would say.

>> Okay. And then uh last question for you are the implementations of your custom RPC methods are they open source? They are not open source and frankly it's at times almost a stop gap to solve for a customer customer needs or teams that do need them until clients um um the binaries themselves implement it and in the case of eenro transaction sync we pretty much just follow the specs. So >> okay perfect please another round of applause here. [applause]