erpc: Solving Ethereum RPC Reliability Challenges - Kasra Khosravi - eRPC

Transcript

Everyone, hope you had an amazing Devcon experience so far. Let's just start it with Do we have something? Oh, there we go. Let's go back. Um, my name is Castra. I am one of the core builders and contributors of ERPC, which is an EVM or Oh yeah sorry. uh which is one of the uh basically EVM RPC proxy tool. Um ERPC has been used in production by a lot of um important web three projects including circle Oiler, Moonwell, Morpho uh even if Pandanda ops team is using

it. So we have we're really lucky to get the feedback and u you know users from all these teams. Um the main topic of this talk is basically like solving EVM RPC reliability issues. Um but it's important to take a step back and understand you know in a really simplified way how does the EVM RPC sack looks like and what are let's say the common failure points. Uh for any web 3 team that wants to start building something on you know blockchain even a small team or a big team. Um there's a really important caveat compared to let's say web two world. um you know from day one you might need to handle terabytes of data if you're on like multi- channel and all those things um you know just the sheer

volume of data that you need to handle is just a lot and just compare that to like a web two company right you know you don't start with terabytes of data from day one you might have some external data that you kind of start with uh but it's not like production critical data that you need to handle so that kind of brings like the importance of like we have a really kind of important job on our hand to make sure that data is handled being handled properly. H and that kind of brings us to the point of RPCs because JSON RPC is kind of awesome. So kind of tries to be a standard for us asking blockchain about data. So just take like different you know use cases on blockchain. We have you know defies they allow you to do stuff you know indexer just get the data

from blockchain and processes it and put it somewhere front end show something to the user and each one of these kind of use cases send different RPC methods uh you know like get give me all the events like e get logs or um you know give me traces debug all those things but the way it works is like each one of those kind of you know use cases send the RPC method to the upstream level which we call let's say our you know um best RPC providers. uh you might have sorry uh you might have like you say your enterprise um you know RPC providers public nodes you know you can even self-host it yourself but inherently because each one of those let's say nodes um have multiple like say infra

layers like we have seen the cloud for like outage like two days ago we know about AWS and each one of those you know providers are working with let's say different clients and all those things so we just have a magnitude are like different failure points. So I'm pretty sure all of you have faced issues as you know as also the previous talk as well that you know different clients just give you like different you know results for you know get get loss for example or um your app is in US east and the RC provider is in Europe there just like added latency um you know there's timeouts there's issues all sorts of things and the users kind of don't care about any of this they just want to you know lend borrow you know they don't want to kind of see the wrong results or have issues

So it's kind of our job to kind of solve those problems and very briefly like each one of those you know providers are working with different um you know clients amazing people from all those teams here but again because also in this layer we have DevOps related we have you know code logic related different versions could have different issues we actually had issue about one month ago that like one of the clients just were like processing the lock index incorrectly and that kind of the downstream effect of it was the user was getting some sort of a you know negative positive balance on you know their app. So again inherently we have u different reliability challenges in this layer and that's where kind of ERPC comes in. ERPC is um oh wow this slide has gone really bad but I think you get the point here.

[laughter] Um that's where ERPC comes in because ERPC is kind of a captain or like a top middleware layer that sits on top of your RPC stack. And if you're a small app, you might have only one node. But as we know like any production company, you need to be on multiple chains. You have different nodes, you have different clients. So just your RPC stacks becomes like complicated definitely. Um so ERPC helps you kind of you know to I mean in short it wants to send every single RPC request to the best node possible and we give you all the knobs possible to kind of you know configure based on your environment. Sometimes you want to increase reliability, right? Sometimes

you want to increase data quality. Sometimes you want to reduce latency or reduce your cost. So you can think of ERPC basically as like is it it's a load balancer. It's a caching tool. It's a data validator. It's a failover system. So for example, if one of your nodes is down, no problem. ERPC just handles it through automatic failover. Or what if one of my nodes is slow? Okay, ERPC has h mechanism. It just sends multiple parallel requests to the nodes and the first response we get is we're going to accept that and we're going to cancel all the others. What if the nodes disagree on a result? Okay, we have consensus policy. We just send the request to multiple nodes and you know if three out of four agree on a result we're going to accept it. So just like increase the resiliency and reliability

of your RPC stack, you can just sleep better at night. Um and also in the past year I have probably talked to hundreds of teams and it's really funny that a lot of teams just build something in house for this and you might ask okay why don't I just use a like traditional LV solution like I know envoy or engineext or something and the point is like none of those tools have EVM context right you just don't want to you don't have like a simple health check like is the node is up or down you might want to have like the latest block high tracking like how far away is behind the tip of the chain or I want these nodes to kind of handle these methods because I know that you they're optimized for it but like I don't know maybe uh traces needs to go

to this method because you know so it's just like EVM knowledge about finality about different methods about different chains uh block height like all these things basically is baked into the ERPC different normalization of errors so you can just basically again route every single RPC request to the best node possible uh now I think it's a really good time to walk through some production examples and we have worked with a lot of teams and you know we Like these issues happen a lot with RPCs and we're going to see how ERPC is going to help with all these issues. So one of the most common ones is like upstream outage like your primary RPC provider just goes down for you know 1 minute 2 minute 5 minute even one day and then okay what happens you

know your indexing stops working your front end you know doesn't show anything to the user a devops person might need to get up at 3:00 a.m. the morning and take out that node and kind of put another one. No zero downtime basically. But VRPC like there's kind of no manual intervention needed like you have automatic failover. Um the way it looks like in the code is and by the way like we tried our best to make the with ERPC you need to have like a you know you can run it like with a TypeScript config or also like a YAML config. And we try to make it as less opined as possible. You can just you know tune it based on your environment. But just imagine you send the RPC requests and um it fails for whatever reason. Also in ERPC we have

knowledge about you know which errors are retribable, which errors are not retriable. It could be like a client issue. There's no point of you know retrying that. But imagine it's RPC request that is actually retribable. So you have like a retry as part of a failover mechanism at the upstream level which is your like quick node anchor whatever it is and then you have like at the network level which is you know arbitron base and mainet. So that request comes it fails on the upstream level and you can configure like how many times you want to retry it. For example here we're going to retry it for two times with some sort of a delay config and you know jeter all those things and imagine it also fails then at the network level we're going to retry it up to four times and then only if all

those four times fail then we're going to return the error kind of back to the user and you can you know configure it based on your environment. Sometimes you have a really hostile environment, you know, using on public nodes or like a really crazy fast chain like, you know, hyper EVMS, you just want to like have the retry a bit more, but sometimes you want it less. So, it's really dependent on your environment. Uh, this one is probably the most important slide because this is one that if it happens to you, you want to pull your hair out and uh it's basically you send this request to multiple nodes and they just give you wrong results, different results.

you know like it get lost for this node just returns four log this one three locks so what happens like you are going to miss events you're going to miss transactions and it's so subtle that you might not even know what happened until a user reports to you like you know hey why is why is that thing is not there uh and the way ERP helps it is like a consensus policy right so I'm just going to give you an example with the consensus policy that we have imagine you send a request to you know ERPC and um upstream three for example is giving you a different hash in that response. Uh so what's going to happen is like because three out of the four agreed on a result we're going to accept that result and then we're going to log that upstream three as like a

misbehavior. Uh and also you can put that misbehavior you know full request responses you in a like S3 or something you can just go back and debug and understand what's going on. You can be really explicit about you know how many participants you want. You might like data quality might be really important for you. Might want to like send a single request to like five nodes and if four out of five kind of agree on the result then accept the result. What if there's a dispute like the nodes don't agree on the result then you know you might want to do certain things. You might want to return the error to the user. You might want to accept the most valid response. What if there's not like you know stand um enough participants?

You might want to like do something else. And some like methods also have like some fields that are not kind of standard. So you might want to ignore them in the consensus policy. So again you have all the kind of configs at your fingerprint to just increase the resiliency of your RPC stack based on your environment. And by the way this config has been really useful for our you know some of the customers on you know fast chains like Hyper EVM mega because there's there's a lot of reliability issues on those chains when it comes to data quality. So this has been super super helpful.

Um, oh by the way, I forgot like this original idea of this was from engineer from the circle team. We were really fortunate to get you know this sparked by them. Another one which is a really common one is a slow responses right and it doesn't happen for other requests sometimes just like my debug methods are a bit slower than others and just there's going to be you know delays and the timeout on other part of your stack. So it just indexing stops working again users are going to have issues and in your PC we have like few ways of handling this. One of them is uh which we call like a hedge feature or a pal request. So you can basically say if a request comes and it passes more than this hardcoded amount of time imagine like 20 millisecond for it send for for

example it get blocked by number for the real time for I only tolerate up to like 10 millisecond. If it goes more than 10 millisecond, I'm going to send to multiple nodes and whoever is going to give me the result fastest, I'm going to accept that. I'm going to cancel all the orders. So that's just going to again increase like the speed of uh your RPC stack. And uh um for now this quantile based feature that we added was really really useful because sometimes you want to adapt based on different you know situations. You don't want to hardcode a certain um um waiting times. Okay, if this request is slower than P95 latency of all the other methods, then I'm going to basically start the hedging mechanism. So that just helps a lot with, you know, increasing reducing the

latency of your RPC request. And this one is probably like the only slide that is not part of the ERPC open source core. It's what we call a edge node and that's like one of our premium features. Uh it kind of combines your RPC nodes. You can have your you know RPC nodes from different providers. you can bring it you know to us and we kind of have a edge node uh service. This kind of get combined together. It can be even deployed to your own you know info stack for just the lowest possible latency. And with that you know this uh edge node is like um uh kind of a proactive indexer that is on tip of the chain. imagine like for the last 10,000 blocks and it gets like the raw data and it can reconstruct like lookup methods

like get block by number, get logs and all those things. So, and also it does like data integrity checks as well and the effect you see basically on the result are like you're going to see like under 10 millisecond real RPC responses after you kind of enable this node and whatever RPC methods it cannot handle through the edge node, it's going to basically handle through your backup RPC providers. So you just get like a benefit of both work. So if it's interesting to you just come talk to me or you know contact me through our telegram and uh the website. All right. So this one is also really really important one. if get logs a lot of different you know application developers we talk to they have issues with different providers because again different providers have different log

ranges right and then um you know you need to manually intervene and do something on the client side because you know uh not all of them are let's say standard but with ERPC we do like auto lock splitting kind of automatically so it kind of um you don't need to worry about it as a developer and the way it works is um uh by the way you can also define like a max allowed range in the ERPC config so you just don't want to you want to um stop the abuse of your SA ERPC instance but imagine like you send a like 20k block range uh it get logs and okay is it less than that 30k that I defined yes it is then for example my node one or node two only support like a 10k range so that request is going to be

split in like two uh like 10k 10k and then we're going to like merge the results and give you the responses and also there's data integrity checks and all those things so the order of the respon is not going to be basically um have issues. So that also helps with the you know the get logs which is a kind of a pain for a lot of you know application developers. All right, last but not least uh I think this one is the one that I'm really proud of about ERPC because it just makes it way more than a like a dumb LV solution because for literally every upstream RPC you have in your stack, you want to dynamically track how is it like performing, right? It's not like traditional elbow like you overwhelm a

note too much and actually because of that overwhelm it's the go is the note isn't going to go down and then you go to another note you want to dynamically always track like how is this note doing in terms of you know multiple characteristics you just don't want to do like a simple round robbing and ERPC kind of lets you do that on like the different dimensions and it's up to you as a developer to understand your environment and see like how do you want to define So for example, imagine you're building like a D5 app and you for the real time you want to punish the nodes that are lagging behind the tip of the chain. So for example, you want to give them like a 20 score in the scoring mechanism, but you don't care too much about the error rate like the block head

lag is the most important thing. But for the finalized data, maybe you want to actually punish a node if they are having a like a large latency, right? Um so ultimately like these different dimensions is going to result in a score and based on that score like the nodes that uh basically have the highest score they're going to get majority of the request and the nodes that are like constantly failing constantly behind the tip of the chain. I mean they're going to still be tried because you know a node could have like a transient issue.

So you might want to still like try them. Um but uh again like this helps a lot by keeping your RPC stack healthy and make sure that you know not a single node gets overwhelmed. Uh I think I can skip this part. Um yeah these were like some of the examples of you know how ERPC is helping with different cases probably in the if you go to our GitHub probably like you know hundreds of you know these different cases that we're handling. Uh but yeah, please check our GitHub, Jo, join our Telegram. There's like really smart people that are having interesting conversations there. And uh thank you [applause] Kazra. What an what an excellent

presentation there. Um we do have time for just one question here. Um and the the question is with well actually a couple of questions here now I see here. Uh the first question is what's the happy path latency overhead of using ERPC? >> Oh that's a really good question. So we have optimized this a lot in the past year but you can expect like some added latency maybe five millisecond or 10 millisecond but again based on the environment if you care more about data quality that kind of latency can be also added more because you're checking across like multiple nodes. But if you mostly care about latency, there are ways of making it like really really minimal basically.

>> Okay, perfect. And uh last question I see here, it says with so many uh config options and use cases uh do you have examples of optimized configs for different use cases? >> Excellent question. We have ERPC basic if you start running it there's a lot of sane defaults already in there. So you don't need to like worry about it. We have some good stuff in the docs also. So if you join telegram you can ask us about you know your environment but because it's like really different about different use cases I think you just you need to talk to us probably to optimize for those things but again the code itself has some defaults the doc has some good examples and you can talk to us if there's any questions >> excellent well please a round of applause for caus That's right.

[applause]