RPC Standards #3

Transcript

Hi everyone. Welcome to the RPC standards meeting number three. I'm going to slow roll this a little bit as folks start to join. My name is Justin Florentine. Uh we are here on a bi-weekly meeting to talk about the Ethereum RPCs. How are they standard? How are they not standard? And what can we do to improve that situation? There should be a link to the agenda posted in the Discord channel for this. I'm also going to post it in the Zoom chat and we can get started. We are recording

this just FYI. We'll be posting a summary likely provided by our artificial intelligence friends after the meeting. All right, it's three after. Let's get started. Okay, let's take a look at our agenda here. Cool. Oh, we got some great input from Sally McFarland. All right, so the first three items on the agenda, oh, four items on the agenda, uh, we have resolving site generation issues.

Um, I have not run into these myself. Is anybody else feeling this pain that light client alludes to in the issue linked on our agenda? Is that the correct issue? The one that's linked is about accessing pruned data. Doesn't look like it's about Yeah, that's that's the context. If you look at the very first comment, he he says, "I started working on this here but wasn't able to get the site generator working." Oh, okay. That's the actual diff um that is failing.

So, that's not great. Um if we're going to That's a lot of friction for somebody trying to fix something. Yeah. And the suggestion was to quote unquote just move to Docsaurus. Um I know a lot of other teams are using docu successfully. Uh we are here on the Basu team. I don't have a ton of insight into what is involved but it's probably worth entertaining. I know we don't use it for Ethereum JS our RPC isn't very well documented to begin with but see the portal networks RPC spec I don't think they use docu source. we use something else and it it seems to work well, but

um Okay, you know, one of those things I don't know a whole lot about dog generation, so just shut up. Same. No, it's all right. Um it's going to be a big blocker to everything else that we do, right? If we can't actually open PRs, validate that they look right and push them forward. So um yeah I mean so just for point of reference the portal network spec uses the open RPC stuff which is like you just you can actually just dump like it generates like a JSON file that you just dump into their playground and it generates a whole spec. So, I'll just put it here in

the chat just for ref and click on it and look. This is what the portal network uses for our RPC spec. But, I mean, you know, it looks the same, but it's super easy to generate stuff. Like, just know I just I just I just know that they have like like I didn't set it up. You I'd have to find the person on the trend team who actually came up with it originally, but it it it works really nice. I don't know about I said I can't I don't know about what has been used on the on the API but that's a a useful point of reference and it's super easy to use and like you can just like you can literally like you can you can generate the updated spec locally from whatever changes you make it just it just dumps out a big JSON blob and you

dump that into the playground and it'll regenerate the spec for you on the fly like in your browser directly if you want you know there so that's just one if y'all wanted another point of dyes totally fine. But if you wanted a point of preference for something that works really well, at least for non docs experts like myself, um, Open PC's playground has been really nice. I that might be the status quo where we're at right now. I'm unclear. Uh, but we should definitely review that. So, I'll add that as an action item.

Um, yeah. So we want to do a three-way comparison between uh JSON IPI status quo and docuurus. Okay, great. Uh moving on. Uh standardizing the you are very clicky my friend. Um sorry about that. Yeah, let me move. Sorry. I also am rocking cherry MX Blues. It's a problem. um standardizing the API to defer to prefer rather decimal integers over hex for all numbers. So this is an interesting one.

Um we've talked about this a couple times. We came to the conclusion that this is a decentsized proposal. Um but we don't have a clear path forward on how to do it. So I think I'm going to sharpen the question today and I'm going to ask should we update all the places in the spec where this is a failure and then add it to a rule after the fact once we have implemented that rule or should we implement it as a rule and have our tooling tell us all the places where it fails and move forward. I think it's six of one, half dozen the other. I'm just

trying to get a proposal moving forward that leads to action. Um, does anybody have any preferences? Sounds like we just want to do it and then move forward. I I just have a quick question. Yes. Uh where are these agenda items coming from? Like who who has uh asked for this and how many people want this?

Oh. Oh, okay. Uh great question. Um they've kind of just filtered in organically through conversations. Um and this particular issue has been on the agenda for I think two or three calls at this point. So it kind of keeps getting kicked forward because uh we don't really have a good action resolution to to put it to bed. Um, so, so to answer your question, they're kind of just organically harvested from the community through conversations on Discord, um, through issues opened into the execution API, uh, repository, a number of different places. Ah, gotcha, that makes sense.

Okay, so now, now the question is how do we move forward with it? People want this, how do we implement it? And the feedback last week was that this is a decentsized proposal. Um, but I I don't know what that means as far as how we move forward. I think generally to me that just means okay, we'll do it anyway and get started sooner rather than later cuz it's going to take some time. But today that would manifest just as opening an issue and then waiting for somebody to pick it up and you know drive it on their their own initiative. Um so that maybe that's the answer.

And does prefer uh decimal over hex mean accept both? I don't think so. Okay. Should probably be worded more um precisely. Yeah. So maybe that's that's the design step, right? We need to write this issue and actually go through all the places where we say hey that is a decimal integer and should be decimal integer only like that is a quantity.

Um, yeah. Okay. Well, that's uh design work that needs to be done. I'll make a note of that. I don't know how much of that I'll be able to do myself, but I'll take a crack at it. All right. Removing aged out fields. That is a PR. This is a very very simple PR. Okay, this is execution APIs poll request number 637. This is also linked to from our agenda. It is a twoline change that removes the total difficulty from the

response from the block object in the examples. Okay, we have approvals from two people on the PR, but nobody that can actually merge this. Does anybody Nobody's on the call that has merge permissions into the execution APIs repo? Is that true? Well, that's a bummer. Okay.

Um, I guess I have to escalate that to all core devs and I'll see if we can find a regular person that has that permission to attend these calls. Okay, we also have another and this might be the same answer. We have another PR uh into the execution APIs pull request 600 which is proposing some error codes. And again, this looks like it was

uh got at least one approver on it who likely does not have merge permissions. There's an ongoing [Music] conversation. But yeah, we have the same problem here. Traction. We're not getting any traction on these. So, anybody have anything to add? I mean, EthereumJS wholeheartedly supports doing this. We actually just removed like I mean, just going back to the previous one, we we removed those um that total difficulty from our because that was a required field in some of our internal stuff and we just cleaned that up recently. So, right, I mean

it it makes perfect sense to do this. But, I mean, if Felix is on board with it, then shouldn't be any holding it back. And Okay, then I can that sounds like cat wrangling. So I can take responsibility for that and get some engagement from committers. Yeah, I mean here again ejs I like here we're not a production client so our our opinions don't count for all that much but we're we would more more than wholeheartedly support this and we do some here again hate to keep bringing up portal network but we just found you know we've we've started standardizing our error codes there too and that's it's very very nice for the crossclient testing and for hive testing and that kind of thing to have standardized error

codes 100% 100% it's a non it should be a non-arguably improvement to high testing and everything else else. Yes, this is a great segue. Thank you. So, if we look at a comment from McFarla on the agenda, we see exactly this. We see issues in the RPC compatible RPC compat rather hive test suite that arguably require clarifications in the specs. Okay.

Um, so we see things like pull request 641. Uh this is a draft PR that's open and it proposes changes to the uh state YAML specifically that a storage proof is not a bytes max 32 rather it is a UI 256. Those things are equivalent. However, one of them allows a leading zero. And so in cases where we need to

allow a leading zero, we prefer UN 256. So I guess this is in draft. It's kind of a request for comments on it. Um, does anybody have any thoughts that immediately come to mind or do we need to have this conversation asynchronously? Again, that's poll request 641. I'll just post it in the Zoom chat to make it easy on folks.

And when she says in 256, she's talking about an actual integer value between 0ero and whatever 2 to the 32 or whatever that is. Is that the implication here? So that's a good question. So these are for uh using for keys keys for storage slots, which I mean I I get that in principle, but like it's easy to say U and 256, but what is the actual representation that she's talking about? I'm looking to see if I understand components 256. I mean, you can have a hex rep. I mean, we return numbers explicitly as hex encoded strings, I think. I mean they're 32 bytes, right, for for storage keys or addresses or whatever in the RPC,

but it just seems like there she's ar she's objecting to to to zero x as an empty value for the keys, which that's that makes sense because your key shouldn't be empty if you're requesting storage. I see what you're saying. So it's it's almost a little orthogonal, right? Well, I mean, I think it's I think it's it's more just like you just like explicitly rule out 0x as a key. Like that should be the disallowed value for storage keys, right? Because you can't have you can't request an empty story. There's that doesn't make any sense to even request that. You can have storage slot zero, but not storage slot 0x where 0x is equivalent to the empty the the empty value. I guess that's what that's what I'm trying I'm just trying to understand here like

Right. So, so maybe there's a different approach that we want to take. Instead of replacing uh bytesmax 32 with unintent 256 um change how we render bytes all bytes max 32s well I mean she's saying okay so that hex encoded 256 bit unsigned integer that's that's a that's a 32 byt or 32 hex encoded bytes I get I don't know I'm just like it and different languages call these things different things. So I think it's just you know different like I don't we don't even have a concept of a bytes 32 max in JavaScript. I we just you know it's just bytes and then we can explicitly say this is this is a 32 byt hex encoded

string you know represented as a hex encoded string but it's going to be 64 characters long because it's 32 by with a zero x on the front. Yeah. I think I think it's just like it like and maybe I'm just ignorant, you know, I don't get down in the weeds on some of the specs sometimes, but like let's explicitly say it's a hex encoded to introduce and she does. I guess that's fair. Um, yeah. So, I mean, go put it on the chair. H pattern.

All right. Feels like it's okay. An alternate suggestion would be bytes max 32 simply does not allow zero x and that would just really just be a change, right? I mean, I'm just trying to under like here again, I think I'm just ignorant of like Sally's like very very succinct proposal. So, if I understand what she's saying, she's just saying it has to be a hex encoded 256 bit unsigned integer, which I agree. A string or a storage key should be or a slot storage slot key should be zero

to two to the 32 or whatever that is. Can't remember but I don't know I yes I I agree who's passing in zero who's passing in empty values for storage slots. But anyway yeah I I think it makes sense now that I've looked at it again. It just she said you hit 256 without explicitly stating it was hex encoded until the body. Okay. Yeah. I I guess it's like not even who's passing it in, but if it's allowed, you know. Yeah. Yeah. Yeah.

Know that makes sense. Yes. And that makes sense to me now that I've wrote I just I I haven't looked at the schema to see why what is the bytes max 32 like that. I wasn't even familiar with that type. But here again, I haven't looked at the I don't look that deeply at the execution API like the sub components that often. Yeah, I mean this makes sense. Uh you can't like hex encode 0x into an integer. So or you're not supposed to. I mean, right, it's very it's very easy to represent it as zero if you're not careful. That's bitten us a few times.

But of course, then you get the whole question should it be 0x0 or 0x0? You know, have to be repatted, right? So the next question is is a UN 256 preserving leading zeros. I mean it's just can you you know do do you do you accept unpadded hex? That's a whole other question. I haven't been part of these conversations in the past so I don't have any idea but um right and I'm pretty sure that like since GU is the kind of the gold standard they accept 0x0 I'm pretty sure for slot zero. I mean I have to check it but I'm pretty sure they accept zero as a 0x0. they don't accept 0x. You're not going to get slot zero if you put 0x as your storage key in the

RPC request, right? But 0x0 should work and 0x0 with you know 63 zeros behind it should work, right? Because I mean you just it's it's an integer at the end of the day. It's an unsigned integer and so you can convert I mean we don't I mean we're not we're not doing EVM math here that requires us to have you know explicitly 32 zeros or nothing and I mean it's can be a little more generous in the RPC since the day we're converting it to a number. So it makes sense to me. I you know um I think I like it the more that I think about it. Yeah, because the UN56 is at

least going to keep it bound. It can be shorter because it's got leading zeros. Uh the 0x is going to be required. So, okay. All right. All right, I'm going to comment to the fact that uh this was dis this how about how about this? This was discussed on RPC call number three and no

major objections were found. We'll continue to solicit feedback. Cool. All right, everybody's cool with that. We can move on to the next John. Uh, hex strings, oddlength strings and representations of zero.

Okay. So, this is this is a representation of 0 and hex. 0x 0 is arguably more correct, but we allow and expect 0x0. Uh, and this is within the context of additional UI 256s.

So, 32 byte words essentially. Um, for those following along at home, here is the definition that is being discussed. Interesting. So, should we allow oddlength un 256 strings? Yes, cuz we we know what we're going to try and cram them into. We know that they're

unsigned. Why Why would we not in an RPC endpoint? Why what I mean here again there there's no functional difference if we're always going to convert it to numbers in ter well I mean people do whatever they want in in in their implementation but I mean it it's representing a numbers are fundamentally not even or odd length they are in when you're do in the in in in the computer they are but we're not talking about that so why would we say you have to for exactly because we're wrapping it in a string I don't know. I find that pretty

compelling. Does anybody else really want to dictate that all numbers are even length? I don't I I think that's sounds like a recipe for disaster because then what do you do? You pad things out with leading zeros when you need to. That would be weird. Yeah, I I don't see an issue with oddlength. Yeah. Going once, going twice our piece.

Okay. The issue of odd length uh 256 encoded strings. was discussed and determined to be a nonissue as odd length numbers are common.

Cool. And I guess that also answers our question of representations of zero in hex. 0x0 versus 0x 0. 0x0 is fine. Cool. Hey, we're at the halfway mark. Does anybody have anything they would like to interject? We got two more things here. So, null versus null versus not found.

Uh this is kind of goes back to the uh error correction I'm sorry the error error handling conversation that we've had a much much broader thing. Well I shouldn't say that because these are not necessarily error cases right um sometimes they are sometimes they're not but what should be the case consistently when the thing you're asking for is not found or doesn't exist. I feel like this is a big one. This this almost feels like a potential holy war.

I mean, don't we return null today? That's what we do for like blocks and transactions if we can't find it. We don't return an error. We just say null. Yeah. I mean, like, who are you going to bring? That is that is inconsistently specified. Um, and there's some examples listed there. Yeah. No, I mean I see that. I'm just like what are we going to break? Like if you suggest like changing it to something else, what are you like is there a material like app layer use case you're going to break tomorrow like if MetaMask doesn't or if all the wallets who use this stuff expect null and you return a text string not found or whatever. I would assume this is a

breaking change and we would have to handle it as such. Well, there's no breaking change if there's no official spec. But my point being that like it's a functionally break it's functionally a breaking change for every wallet potentially if the if if they aren't, you know, graciously or gracefully handling unspecified, you know, responses that don't match a standard block or something, you know. So if we started specifying it, I assume

things would break. You know, I mean for a parallel point of reference here again, the portal network returns zero X for all of our like lower level RPC calls. If we don't find stuff, I mean our whole network is built around find not kind of things. We always return zero X if we can't find something. We don't we don't return errors or not found, you know, funny special strings, right? Because it doesn't seem like an error case. It's a failed search is not an error. So, but I apologize. I actually have to jump to a portal network call, but this has been fun. So, I'll be back.

Um, I I do think Go ahead. I'll I'll catch up with you later. Um, sounds good. The Yeah, so the null versus not found thing, maybe we should put a pin in this, broaden the discussion, and solicit more feedback. Okay. So yeah, how widely is this a problem? So null versus not found. Next step would be uh determine how widely this is a problem and um propose uh how to

more explicitly specify these cases. Okay. All right. And moving on. Total difficulty. This is an interesting one. So, if you get a block that's prior to the merge, it should actually have something useful in total difficulty.

And if you get a block postmerge, it wouldn't have anything. Okay, it should have a empty field. So I think the question is is do we want to have that as an optional field that we stop returning to clients if they're requesting a block that's after the merge or is that too much magic. Right? So maybe your maybe your schema should not change just because your input values changed. That's a very fair argument.

for this uh API design. On the other hand, it's useless data that's always the same. Uh you could specify that after the merge, total terminal total difficulty is always the total difficulty that the chain achieved once it got up to the merge. And then you just have a static value that's kind of returned all the time. Uh that also feels kind of useless. Um and then another option could be well versioning right maybe we introduce some versioning and there is a postmerge. Oh hash is affected by

it. Are we still hashing over total difficulty? That would be wild. Okay, that should be a Oh. All right, maybe we should investigate that to see if it is truly um useless.

Um just difficulty. Okay. So I guess the question is is like would you introduce a versioned API for pre and postmerge? I mean I think that this would cover a lot of things, right? There's a lot of things that return blocks. It would at least be all block APIs. So this is interesting from a well what is our path forward perspective. How do we refactor this

spec to get rid of something that's kind of crusty? And so if our options are version the API, adding version numbers or allowing total difficulty to be optional and essentially having it change the responses schema. Well, is it a change in response schema? That's the next question, right?

Because it is JSON. So you could just say, well, it's not there anymore. And that should be within the schema as much as we have a schema. But it's also kind of a magic behavior that you need to document and present to your clients. Like they would need to know that, oh yeah, that field might not be there if I was requesting prior to the merge blocks and now I'm not. So, I think I'm personally talked myself into being into the camp where we just make it

optional. I'd love to hear other people's thoughts. Yeah, I think optional makes sense as well, but um it would be nice to have more voices in the conversation. I feel like cuz this could be pretty uh split. I think so. We want appdevs and wallet devs. So, how about this? We'd like to make total difficulty optional in responses, but will not propose any

changes till we hear from wallet and DAP devs. Yeah, I think that's reasonable. It seems uh relatively easy to implement u removal of this field post merge by every client. So might be more like a rule to exclude this field instead of uh mention it as an optional I think. So just a personal

opinion I guess. Okay. Well optional implies premerge right? Sometimes it's there. Like if it's premerge it's there. If it's not postmerge it's not there. That would be how I Yeah, I'll clarify that in the comment present in the JSON for pre-merge blocks missing postmerge.

cool. All right, last thing that we have here is looks like somebody ran spell check and spec check and got a bunch of errors. sorry. Where are we looking now? Sorry, this is comment number three. So comment number two in the agenda here which has a link to PR in uh no an issue a newly opened issue into the execution

APIs repo number 642. Yep that's it. Thank you Carrie. This is kind of aligned with like client's complaint that they couldn't build the uh the whole you know couldn't build locally. File not found a spell. Okay, I guess we don't really need to debug this on the call.

Okay, well that's it for the agenda folks. The floor is open. Anybody have anything else they want to talk about? Uh do we discuss SSD uh API here or not? SSZ. Yep. Yep. uh JSON RPC API is a topic, right? But uh maybe at some point we will try SSD as a binary encoding for requests and responses, right?

Oh. Um gosh, I mean maybe this is really more for the the specs than how do we specify the execution API and how do we um enforce that specification. So right now we don't have uh a lot of this stuff as tight as the consensus layer has it with their REST APIs. um our schemas have you know inconsistencies in them etc etc. So I think all of those things are uh our priority. Um clearly the door would be open uh for changing the encoding format. Is that what you're

talking about? You're talking about the open API the sorry open JSON RPC spec just not encoded as JSON encoded as SSC. Is that what I'm hearing? Uh, yep. Yeah. Okay. Um, I think I'd be happy to put that on the agenda for a future date. Um, but I think we're probably pretty far from that right now. What Why would you use SSD instead of just like gzip or something like that?

Uh, instead of what? instead of gzip or some other transport level compression or protocol like well I would try it because uh the data in such format uh doesn't even need to be compressed I guess and uh uh you can extract uh like objects from the encoding and store them without transformation. But JSON is a uh necessary serialization, decentralization, right?

Yeah. Okay. Um I would encourage maybe to put this on a future agenda. Um solicit some opinions and maybe a proposed design uh from the community. Um, me personally, I'm not super concerned about the encoding between the clients since it's just, you know, your consensus, at least for the engine API, your consensus layer. Um, I I shouldn't say that because I'm not considering wallet devs uh or DAP devs, but those would be the people, I guess, that would advocate for a different encoding format for the engine

APIs. Um, one thing you could also think about, maybe propose a design for is like, well, how would clients specify an encoding format, right? Maybe you have default JSON encoding u but then they set a mime type on their request and they get it back in SS. That could be a good place to start um before introducing uh different encodings. Yeah. Uh uh there was uh some discussion in the discord and uh proposed solution post was to uh use accept header. Yeah, makes

sense. Interesting. Okay, thanks. It's not the topic for today, I guess. Okay, I got my gears turning though. Interesting. Very interesting. Yeah, cuz I guess a bunch of wallets and and dab devs would already have the SSC lives that they need on board. They have schema checking. They'd have all the hashing and all the merkelization that you get for free with SSD. It's an interesting idea. Cool. Yep.

All right. Well, uh, if that's it, we can wrap 12 minutes early. Uh, I will I think the recording will get posted. An AI summary will come out next and I've captured our kind of decisions and next actions in the uh, agenda. So, thanks everybody and I'll see you on Discord.