RPC Standards #9

Transcript

Good afternoon everybody. Um welcome to RPC standard call 9. I think the first thing we have on our agenda is the spell check decision um discussion and the updates from a course 3 if you don't mind. >> Yeah I mean so the the main update as far as the the docs um update is I'm kind of waiting on light client at this point. I've I he I know he I mean all the code is done. It it builds it. You it should deploy successfully with the the CI. I've tested it on my fork and um I'm honestly not sure. Well, he said he said he had some issue building it. He

hasn't replied back as to what that issue might be. So I I can either confirm or deny that there is still an issue of any sort. But otherwise, as far as I know, I think I think Zayn has reviewed the docs. I've looked at them. Yeah, go ahead then. >> My client to see if he still had an issue, but he said that he hadn't had a chance to look to see like okay if >> this is still there or not. >> Okay. Yeah. I mean, I've had the CI set up so that it runs on all the currently active versions of Node. So, it if there was an issue building, it seems like CI would have caught it. Um there.

>> Okay. But you did mention you did mention about the spec check. >> Oh yes. Okay. So that's a second. Yes. So that's that's kind of a sec a whole second um piece of work. So um so I've been talking with Lifeline about this off and on for the last two months. And at one point he thought there might be some issues with um spec check not accurately or generating false positives where it was identifying false fields that shouldn't exist as being okay or anyway so some things he thought were the spec wasn't matching the test and that he thought spec check was check was missing. We've gone back and looked at the the things he thought were issues and they've turned out to not be as far

as we can tell. So spec check should work fine. Um the current broken test he has a fix for there's a PR outstanding for it to fix the E simulate v1 um test that's currently in the that's causing the test to fail on the CI. Um so he just needs to merge that or get field or somebody else like obviously I'm not a I'm not a maintainer so I can't emerge or approve that. Um but that test should be fixed and then I am working on an additional component which is like a it's essentially it does the same thing as spec check but it it it allows spec check if if you have a a test with a broken if you have a broken test or

where a schema validation is failing. It will allow it'll generate a a little URL to where you can run it'll it'll throw it up in a browser window that will allow you to visualize the both the schema and the the test that's failing locally. Um, and it's it's not 100% done. I I still have to make some more tweaks to it. I need to talk to him about how he wants to deploy that in conjunction with spec check. But right now the way it works is you run spec check. If a test fails, it will generate a URL for like this this local kind of like schema viewer. Um, and then and the schema viewer is just using it's it's doing spec validation as well. It's just using a JavaScript version of the same thing is what spec check does. And it

just but just visualize it makes it easier to actually look at the schema that you're testing. So in this case it it might be like an individual RPC method and then you you put you you can copy and paste in the data that from the test and it will tell you what what's failing and then you can use that to kind of triage failing test. So it's it's just like a it's just like a nice DX improvement for people who are working on um either creating new specs or fixing tests or adding tests or whatever it is. So, um, >> okay. Sorry, if you don't mind, I have a question. Does it like interact with the RPC test geno?

>> Um, it at the moment I just have an independent repo that I'm working on, but I I need to ask Lyline how he wants to integrate the two because um he had suggested like there's there's all sorts of like JSON schema validators out there like this. I just I just used, you know, AI to generate to create one. I've done some tweaking on it, but um it it's just the idea is it just it's just because spec check doesn't really produce nice output when you have schema failures or data data validation failures. Um, it's just to make it easier to validate to fix figure out what's wrong when you have a schema. If you have a if your schema is not doesn't match um the way that they're supposed to be formatted or

if the the actual JSON data in your test doesn't match the doesn't match the schema. This it just makes it a lot easier to to triage that like in a v kind of visually if that makes sense. So, it's an additional tool that you could use alongside either the open either the execution APIs tests or the spec check generally. >> Okay. Um so I dropped a link because before I realized that um there was a spec check on the repo. I was trying to like work on this too.

But then when I realized that we have something that fits more into this category, I had to pause. So is it something similar to what you're currently doing? Sorry for the noisy background. Just how the dog is barking. Sorry, did you hear me? >> I can hear you. Um, but I Okay, I sit here. >> Yeah, acolyte, we can't hear you. You're like muted or silent?

You're not muted, but you're not saying anything either, or we can't hear you if you're saying something. But I just wanted to ask, is the the um the two you're working out something similar to this? I I think I the link I shared earlier. you can just try and drop out and then coming back again to see if it works now. Okay, one second. Be right back. >> Oh, yeah. It's working.

>> Oh, is it? Okay. I don't know. I don't know what's going on. Sorry. It's network connectivity or something. Um, so I I haven't looked in detail at your project. Sorry, I'm on mobile. My my laptop is having connectivity issues for some reason. But um it it just from just briefly skimming your project plan, it looks similar. I mean what I was building is really just it was really only supposed to be used in conjunction with spec test and just when spec test um fails uh when a test fails or whatever it would just allow you to visualize the um the error where the error is in the particular schema you're you're validating against since spec check output is not very pleasing to the eye and requires some work to get to

where it's actually comprehensible as to what's going on. >> Yeah. And it's kind of like hidden. It's hidden. You need to like call it before it actually comes out. >> Right. So this this is this this is this just extends what spec check does. So when a test fails, it generates a little local URL and then if as long as you're running this little schema validator client that I built um it just will it will populate it it it populates the the schema and the data and then highlights where the um the validation failures are. So it it's it can be used on its own, but it's not as I've currently built it. It's not it's not a full schema. It doesn't take the entire open RPC document and then parse it. It's just taking the particular schema that

you have an issue with um that's generated from spec because spec already does all the work already and we don't have there's I don't like client didn't feel like there at least my impression was from our brief conversation he didn't feel like there was a need to re to rebuild that entire thing since it already worked. Um but here again it's not rocket science. I mean AI can do in about 10 minutes and get you a basic schema validation tool. So >> yeah. >> Okay. So do you want to say anything? >> Yeah. So I was just like sort of thinking about things um because uh at open RPC we' started working on uh Golang tools to do sort of that same like validation. So when I took a look

at spec check um in my opinion the issue is like kind of twofold. So one uh we have a PR out for a llinter and the llinter is written in Golang and the original e simulate v1 like errors are actually caused by like um just a bad specification. I believe there's a field um called calls which is being required but at the same time um isn't defined or specified within properties uh and that causes the CI to then just error and then to figure out that that's

the error in itself from the validation is incredibly difficult to to read like like you all were saying. Um but that's something that like I think u this llinter project that we're working on would uh catch. So it's kind of interesting to think about this from that perspective. And and then the other thing is is that um open RPC does have validation stuff. So within the TypeScript script ecosystem there's schema utils which I think is a lot better about telling you like where the error is. um and then that would actually do like real validation. Um so one of the limitations with uh the

schema validation as is that it doesn't necessarily always uh um guarantee that the document itself is well formed. Uh and then there's certain sort of rules around reference references and things like that. Um so part of our thinking is that on the road map for this upcoming year would be to uh move schema utils things into more native languages. And so that would allow um the spec check to then use schema validation tools internally. So it would just be able to validate the document and then get back something nice. That's like the idea um and the thinking. So it would be

interesting to see what lians thinks about that. And then the other thing is is that looking at the spec um in the current PR there's uh a little bit of a questionable thing. So right now um technically we've limited the scope of the spec to support uh JSON schema draft 7 but uh light clients PR actually shows for unevaluated properties which is nspec 2019 2020. So it's technically not supported but kind of is supported by the underlying schema validation. Um, so there could be some things there and so

it just kind of needs to be looked at. I'm getting ready to put in a comment on the PR. Um, so that was a lot and I kind of just jumped through some topics, but um, that's just kind of like my high level thinking and like I 100% agree with both of you that like spec check needs um, just like a better way to inspect what's going wrong with it. Uh it would be interesting to think about sort of what that looks like um from like desiraability category. The other thing with spec check is uh that the parameters uh there's like this kind of testing. So

the other reason why at E7v1 is like I think broken is because um when the test came in for the io there are also just standard JSON RPC error things like oh you didn't specify all of the parameters. So if someone sends in like empty parameters you kind of want this negative test where like bad parameters returns back bad results. But the way that spec check does the like parameter validation is such that it will throw because um the RPC request is not made correctly. Uh so you kind of want to do two tests. Uh you want to do one where it's oh go ahead Mercy.

So I also want us to like point out um this general thought because we know we all have the idea that the RPC test gen effect whatever um output it gets gets directly injected into execution spec. I think this is where the whole issue is coming from. We need to like find a way to make sure that whatever we're generating from the RPC test gen side does not ne negatively affect the execution API report cuz this is where the whole um it simulates block states calls issue array and object issue started from it. We need to run some check that it will run it will run correctly on both RPC test and then run

correctly on execution API test. Not in the sense that if it passes for example the it simulates it passed on um RPC test then there was a an object area issue on execution API test. So I don't know where or how the whole thing is programmed but it's also something we need to look into so that we don't create a test that passes in one end and then the other end it fails. I don't know if you understand what I'm saying. >> Yeah. Yeah. I I I think we we get that for sure. Um that like essentially the validation RPC test gen which I think is fine. I think the real problem was that

um the spec check worked as expected, but like I think the external pressure was that the developer that merged in the change like just needed to actually land the change and then debugging the actual underlying issue was complicated and like uh just was like maybe just too difficult. the bar was too high to figure out why it didn't work in the other repo. But >> I don't I don't sorry I don't really think that was the case because they had two options is either they changed the object to array which light client already has done or better still the

batch the cause because the whole issue was from the state um the block state cause it was either the cuz when I noticed the error I I this was the two options that was very valuable at the point then in my mind I was like changing the schema to accept an array will be like too extensive because I don't really know the idea of us having an object instead of an array or in a case whereby we can just have the two in case of any of the test any of the test we're carrying out requires either an object or an array then it should pass.

So um >> it's a problem that shouldn't have happened initially. >> Actually I I looked at the code and then I also looked at the spec and the issue was like that the spec was malformed. So um within the original PR that was merged with the broken code uh the issue was that um when the person went to write the uh EFS spec for the EV simulate they had a required field for calls but then they didn't actually add calls to the properties of the object um

which is actually broken. So spec check threw an error that said that um like the type was invalid and then uh so that's like the error that like got thrown up. Uh and if you just add calls to properties on the object that's being specified in the original EV1, it like fixes that error. The second issue was that um uh they wanted to do a negative test the person who made the the call and the problem with spec check is that negative tests are not supported. Um so spec check will throw an error if you give it

bad uh params. So uh when the test was generated uh the RBC test generates the proper io file um which is like throwback a negative 3200 uh error code and that gets stored in the io right so then thatio gets copied into the espec repo and then when spec tries to run it won't run because the request that is being requested to generate is invalid by um JSON schema validation. So the spec

check schema validation prevents that from passing. So there are three errors in the um and the spec that are very preventable that spec check actually caught and catches all three once you resolve each error. But the problem was is that all three of those were ignored because it's difficult to debug and like um as the person who like wrote the open RPC um call I am very very confident that like that was the issue and I actually fixed it in a local build. Um so I can always like share that sense. We just saying

the same thing. That's what I was saying that we can we need to like build a plan that we don't like for example what you just said the test passed the whole thing passed on RPC it generated a negative call then looking at execution API side it doesn't accept negatives so we need to like make a plan that in case you're writing a test that will generate a negative call it should flag to you so that you will know in you have like that firstand information that okay this is not going to work because whatever you generate here is going to be injected to this side.

>> So it kind of like saves time. It's it's it's a misunderstanding clearly because I think I don't think they actually know that we don't generate negative calls. >> Yeah, I think that that's a bug with spec check actually. So like spec check itself should support um being able to turn on parameter validation >> because there are cases like when you're writing the test itself you want to make sure that you're calling the right um thing according to the spec and then there's a chance that you want to turn it off as well because you can't just like pass um because you want negative validation test. Uh so like basically spec check

needs an arg that changes the uh invalid params to warn uh so that way nci when you're doing this test it works when you're doing your local development you know that you're testing the thing that you wanted to test uh and you have like both sets of information uh if that makes sense. >> Yeah it does. So I also have some little updates on um excuse me Killer had um um APR up. So we talked about how it's going to get merged after life clients PR gets in

eventually and I don't know if we have any other things to discuss. Acolytes do you have? Nothing else for me except I'm I'm very interested to hear for the what Zayn's doing as far as schema validation goes. It's kind of more visual because like that's that's just a super helpful thing with spec check if we did and I I think what's hard is like we don't have any decision makers here >> and this is just like the same the same song every every two weeks and >> every few weeks and we actually need to like get it done and then move forward from here because this is like an endless circle. He has something too.

>> Yeah. I mean, we just need >> we need a a maintainer for these for the execution APIs and it's not going to be me because I'm not gonna be, you know, I don't work for the foundation. So, um, but y'all need somebody who can like maintain this stuff, who's willing to like come to these calls and help, who's actually interested in this because there's a lot to do. It's just not >> Yeah, >> that's me. That's me. >> Um, yeah. So, I need to talk to light client. Um, I'm Carrie. I work for the steel team. Um, and I've been with the foundation for a long time. But, um, yeah, we're it's the plan to to have I think someone from the steel team step

in >> and I think that's going to be me. I just need to like client about getting >> Okay. I was actually >> volunteering to do that. >> Okay. >> Yeah, we're interested in supporting. So, >> I was actually involing to do that, but no problem. >> Yeah. Either way. >> Okay, that's good to know. Carrie, I don't know if you can DM me on the via the E R&D discord um same username with like me and maybe like Justin Florentine and >> and like just so I can connect with you because I I'm just curious to like understand like who's doing what because I have some other questions related to a grant I'm doing for some of this work.

So, if you're actively engaged, like Line just is busy. Yes. Yeah. Totally. Yeah. And I should be able to >> attend the calls and stuff from here on out. So, >> cool. Great. >> Yep. >> Okay. Thank you. Um, but do you have any other things to tell us and then we can wrap up on then? >> Hello, Clary. Wait, I didn't I didn't Were you Who were you talking to? Mercy. Sorry. I didn't understand that.

>> Yeah. >> What did you need? >> Yeah. I was asking, do you have any other information or how we can go up from here since you're the one pioneering this now? >> No, I mean it sounds like the biggest things that we need right now are just merge access. Um >> yeah, >> and like the ability to review. Um but is that different or what are the what are the biggest things that we are blocked on or need help with right now? That should be um the pers we have up and then the to do um website upgrade.

>> Cool. >> Sounds good. Yeah, I'll plan to talk to like client this week and and just get kind of more up to speed on that front. >> Okay, great. >> Thank you. Um I don't know if you have some time you can get me if you can give me a feedback on the link I shared earlier. I don't know if you still have it. >> If if you'll drop it in the I'm getting ready. I have to actually drop off here in just a minute. If you can just drop it in the in the in our RPC channel in Ether R&D. I'll take a look again and um and I'll I'll post my repo that I'd started working on to kind of extend spec check and we can maybe we can collaborate there some.

>> Okay. All right. I think is there another thing we want to discuss or something I'm missing? Okay, guess that's all for today. All right, thank you very much for joining today's call and for your impute. See you guys next two weeks. here.