Interop Testing #33

Transcript

[Music] Recording in progress. Awesome. Um yeah. Hello everyone. This is uh interrupt testing call number 33. This is April 14th. Um we are discussing today um spectra testing in general interrupt testing. Um I'm leading today. So yeah, let's get just started. Uh there are three agenda items today. Uh vector testing, pure das testing and euf testing. And there are a couple of comments in the BM repo that we should go into.

Uh but yeah, I guess we can start with testing um discussion. I can go over just basically uh the the notes in the PM repo right now, but does anybody else have any status update that we want to give uh about Spectra? At the moment, I'm still currently just testing the M workflow locally. Uh hopefully by the end of the week, I will be able to build some petrol blocks on crytosis.

Cool. Is there any um any other details about this? Um are you testing using uh me boost or Yeah, I'm I'm using MV boost and I'm using uh our builder because the gap builder has been deprecated. So our builder with the uh flashbots map 3D and uh their map builder and also after that once that's sorted then I will uh begin testing for Yes, Maurice.

Um, so I went over basically I went over all of the tests for PCRA. Um, because I had a feeling and a few others maybe also had a feeling that we are kind of like under tested for PCRA. Um, so I went over uh all of this again and uh did another round of basically review of the testing uh both on EL and CL. Um I found a few things. I think in total it was 23 edge cases that I found both on the L and CL.

Um but they are being addressed and I'm pretty sure none of them will uh will create any sort of issues. Um but uh with that I am pretty confident about testing. Um and the only thing that I'm like I'm currently working on some tests to uh to do um uh transaction pool testing uh because that is something that I feel is also uh kind of uh missing in general from our testing pipeline.

Uh so uh I'm I built well I'm I'm currently working on some uh some kind of basic scripts that just do a bunch of uh bunch of these 7702 edge cases uh blob um plop fuzzing plop testing uh things just to see that um that all of the pools are are kind of healthy and don't end up um uh crashing and burning in in in these uh kind of edge case scenarios. Yeah, that's it.

Of the I Yeah, I just want to circle back to the edge cases. Um I'm assuming we're working on the implementation of the test cases. Um or do you mean you are uh writing tests right now or I'm guessing that the east team we should we should do it right or what's what's the angle? Well, I think they are. I I don't know if you looked in into our G and East uh discord group, but I think Spencer is writing a bunch of them uh now and on the on the consensus layer. Um uh Justin has already pushed uh a bunch of them and I think is working on more.

Awesome. Awesome. I think once they are um once they're written I think we should like run it against all the all the clients and just to make sure that there's nothing there. Um yeah and yeah uh Justin shared the the pull request where he's uh implementing tests. we should do the same for for for east. Um in general uh one side note that I wanted to bring up is that we are working on at least on the east side we're working on uh postmortems for testing. So any test that we that is found that was not found by the original

uh tests that were written we should like do a write up uh about that. We haven't done it in the past, but we're going to start to do it. And this should include all the edge cases that you mentioned, Mario, to see why there were why why were they missed and what we can do in the repository to improve not missing them or similar test cases, not missing them again. Um, but that's that's work in progress. Yeah, thank you. Thank you for taking a look at the at tests. All right. Uh any comments on this? Does anybody have any uh questions about how um we're going to run this uh this

updated test or any other questions? So will you about to provide a compiled test so we can run against because I don't see the minimal tests attached to it. So what's the plan there? Uh yeah I can I can try to well I'll share these with the clients after merging it. I'll initiate like a nightly build of the spec tests and then we can run them with that.

Cool. Thanks. On the E side, we should prepare a another release for that for the test to um it's probably going to take uh until the end of the week until we we're able to to release, but we will make sure that to run it with every client on Hive beforehand. Yeah. Any other question regarding the specs testing?

I have a go. Oh, sorry. Go for it, Justin. Oh, uh, sorry. I was going to ask if clients uh have the ability if all CR clients have the ability to run nightly spec tests. I I know a couple do, but I'm not sure if all do. Is there anyone that doesn't have the ability to do this? Okay. Terren says we don't, but we can support.

Okay. Yeah, we can talk more about that later, I guess. Thanks. But you want to talk about it now? It's fine. Anyway, yes, sorry then. Oh, yeah, no worries. Um, I just wanted to add that, um, Marius's comments have been added to an issue by Spencer. Um, it's in the Stealth repository at the moment, so it's not entirely public. Um, just while we go through it all. Um, and we realize that it's quite last minute for more releases for client teams.

We'll be running the tests against all the clients ourselves. Um, and if we find anything, we would raise it. But as Marius said, um, he doesn't expect any of these things to trigger any, uh, unexpected behavior, but we'll keep an eye on it. Awesome. Um, all right. I guess we can move the move on. There's two topics in the PM repo. One was raised by Spencer.

Um this is uh regarding the new modified uh withdrawal contract uh tests. I think they they were causing a lot of issues um in some of the clients, but maybe they were test uh test case issues. Um sorry, Spencer, if you can uh expand up a a little bit on this uh anything that you wanted to share. Yeah, I I can go quickly. So um some of you know that um within the testing team we've been reviewing the coverage over all the EIPs and um one of our new um team members Felix um wrote wrote these tests which essentially

u modify the maximum allowed withdrawals within a withdrawals request um and modify the code in the contract. So it's something which um in theory would never happen, right? But um it's still an interesting case. So I say this or preface this first that it's not critical. Um but yeah, this was flagged in the interrupt channel last week from Basu um because or at least my understanding is that there's confusion um that the uh system contract changes um have added confusion here. So essentially with these new new test

cases we have from my understanding um Basu and Reth when they call the modified um contract code um they correctly produce an invalid block based on the new changes um to the system contracts. But GU and Aragon um correctly call the contract codes. Um and um so sorry, Bezu and Wrath produce an invalid block. Um and G and Aragon produce a valid block, but based on my understanding, it's um what's happening is both correct on both sides. Um and the key difference is that Basu and RET

call the contract code and it triggers a stack underflow within them. um GU and Aragon called the contract code and they don't trigger any errors. They call it correctly. So, I guess I wanted to bring up um to see if clients uh care about looking deeper and whether they the call that's causing an error actually matters because it's not something that would happen um in practice. But um if clients do care, but looking deeper, it's uh something that could make things slightly more robust. Um I'm aware that it's not the highest priority, but if um clients care. Yeah.

Anyone from from Bessu or Ref or G or everyone that want to uh comment? I I can just say from from Besso side, I mean we we think it's it's just you know that's that's a very theoretical edge case and I mean we can of course make changes but we just we should just write down in the spec how this case should be handled. I I think you know then we we can do that. I mean as as as Spencer said, I mean it's not urgent because the current contracts cannot change but yeah for for for the future in case this somehow happens I mean it

would be best to just clarify it really in the spec and I I don't know I I I I don't really have a strong opinion which is the correct case I would say. I I guess I'd also add it'd be interested to know why um bas it triggers a stack underflow with basian ref and not with the other clients. That's enough.

Yeah, I I I still need to look uh into it uh exactly, but I'm I'm not sure. uh from Aragon I could just tell you that uh the modified behavior for the EIP change hasn't yet been merged. So if indeed there was uh an issue because of the modified contract, Aragon would just uh silently ignore it and go about its business producing the block.

I guess I guess I can expand a little bit on the motivation on why this test case was produced. Basically the idea is that it doesn't matter um what what the contract give gives us. It can I mean it it can it cannot fail as as it is. But what happens if it fails? Uh I guess the the purpose of the test case is to have all clients aligned so they have the same response. Um whatever this extraneous behavior, um it's really unlikely that we we run into something like this, but if we if it was the case that we ever run into something like this, um that we should like handle it the same way. Um but yeah, I I'm I'm not

um I don't I don't I don't understand right right now the what happened on the consumption of the test, but I can take a a deeper look to see if we can uh find find the culprit. Yep. Any other comments regarding this? I guess the the last thing I'd add is yeah, if Basu and Reth are interested in looking at why the stack underflow occurs and they have the time, then that would be amazing. If not, um it's not a big deal and we can skip these tests in the next release.

Yeah, we we will definitely look into this. Also another mind is mentioning uh if there were not uh executed with with it or did they pass or what was the if if there's no no response now I think uh we can we can just uh take a look later but but yeah I think we should run with dynamite that's what I mean yeah I didn't look deep into it but it looked like a slightly different response um but nothing nothing major okay All right, moving on. Uh, the next topic, um, this this is something I erased on the on the PM repo. So,

basically, I'm I'm going to share the link to the to the comment. Um, I just searched in the chat. Um yeah, this is uh we found a couple of issues uh last week by inspecting the exceptions that every single execution client gave us when running the vector tests. There were some um some exceptions that caught our eye and basically we found out that they were not hitting the expected um handler exception or or or invalid case handler.

So in general we have this set of invalid block tests that we run and basically when we run them in hive we just expect the client for to reject the block but in these specific cases they were rejecting because there was a mismatch in the receipts or a mismatch in the gas used or a mismatch in the stage route. Um, I'm going to put a link to a little bit uh to to a deeper analysis on this um and the reasons why we did it. But yeah, basically it boils down to um do we want to make this a consistent check. Um what I mean is that we do we want for Hive to check the to be strict on the on the

exceptions that it that it uh that it verifies not only pass the test if there was a rejection. Um and and the the issues that we found last week is that this is this is required this is needed in some to some degree for us to catch these tests. The downside is that um one we have to maintain a list of exceptions an exception mapper for every single client. So for example right now we have a PR in east that adds an exception mapper for for Go Ethereum one for Nethermine and so on and so forth. And we h we will have we will we will have to update this uh when we write new tests that generate more kinds of

exceptions. And also if the client for some reason modifies the messages the error messages that they they they return when some invalidation occurs, it will also generate now a a um an a failure in hive tests. not because there was an an incorrect behavior but rather because the exception just changed. So all in all I think um my question to the client teams is this something that um um do do does everyone everyone here seem like is is something that we are capable of like dealing with is is it going to be an

issue? I mean from east side what I can say is that we can try to maintain this this list this is exception mapping list up to date. Uh one option is that we always maintain it to to up to date and we always run with this strict exception mapping and the other the other option is that we disable it and we only enable it it once we are like deeper in the fork cycle. Uh so we we see less file positives and you guys are like free more free to to modify the exceptions if you have any modifications to do. Um yeah my I think I think overall my my my my question to client teams right now is do you often modify the exceptions

strings? Is this a problem that you see um that we maintain such list? Yeah Lucas. Oh, generally we don't really modify the exception strings often, almost never. But if we want some uh hard mapping, maybe uh for at least for those errors that we we want to test for, we should create some kind of error code in payload status and just define it in spec.

Yep. Absolutely. Um, in my opinion, the place to to have an exception mapping or exception list that is canonical is yields and we should have it there and just rely that to see um yeah, I think I think we should we should have this list and also use it in yeast. Um, yeah. So, Dan mentions that um yeah, it brings exceptions into the protocol. Um, I agree. Yeah. Yeah. I agree that this is is a protocol change. Um um my my argument is that I mean we don't have

to make it part of the protocol. Maybe just part of testing and we just have a lot of care of um updating these lists. Yeah. So yeah to expand on that I mean there's some situations where there's two things going wrong and we have to choose which error string is more important or we have to encode that both error strings are acceptable. Um a classic example would be if um a log is wrong. Not only will the log be incorrect but the root hash will be incorrect and for performance reasons a lot of clients will just look at the root hash and not even consider the rest of it and move on.

uh this could be handled in a test and still be like uh that it needs to return one of the errors, right? And we could just specify the test like that. Yeah, absolutely. That's exactly what we do now. So, for example, for the requests hash uh exceptions there were some clients that were rejecting because of the the block hash was incorrect and the others were rejecting because the request hash wasn't correct. This is encoded in the test. the test suspects either of those exceptions because it understands that the logic can be different. So I don't think it's going to be a problem for us. We haven't already been doing uh so we can just keep on accepting multiple uh multiple exceptions for for very specific tests.

Um, yep. Any uh counterpoints that anyone wants to raise um or concerns? I I I would be very interested in in hearing those. So if we want to continue what's been done here like parsing the uh the exception strings etc. It would be really good for the client teams to know which exceptions are you relying on and to the even to the point that if

they are changing it uh they should know that they might just submit a PR there for the parsing right that's a great idea yeah I can share um I'm going to share it on the discord after after this call um we basically have five different files And it's it's just a very very simple Python uh dictionary. So it should be very to be honest and never mind. I would just find where this error string is being what what I want plan to do uh find where this error string is being uh created uh or used and just put a link there. If

you update this, update that and put a link to the mapper. And if we do that in potentially even all the all the like in a comment right or or something uh if we make that we can kind of link it so it will be linked in a way that's yeah absolutely all right. Um I'm going to I'm going to share uh after this call links to each mapper that we're using. just just just just for everyone to be aware that uh uh where where their their exceptions are. Yep.

Uh thanks and uh Dan just shared a an example. This is this example is for um this is for grow ethereum and uh yes as you can see it's basically just a dictionary and um and yeah it contains it contains all all exceptions that are expected from goer. All right. Um, if there are no more comments, uh, we can move on. Um, the next item is, uh, give me a second. Is spear dust testing? Um, oh, let Okay, let me go

back. Uh, is there any any anything else that we want to discuss regarding spectra? Any other topics? Um, I'd just like to add, um, just in case anyone missed it, it's very easy to configure the client version that's tested in the Hive spectral dashboard that's currently only really relevant for the EL. So, it's just a few links there. I'll add it to the issue as well. So, as you create your release branch, you're very welcome to um, specify that as the version tested that you can see in this dashboard. Otherwise, just leave it as master, whatever suits you best.

I guess one remaining for Pectra would be that I think we decided that we will will do a testation performance analysis uh once all stable releases are out or at least maturity so we can get the proper final analysis unless someone thinks it might be good to do another analysis uh today or this week. So just based on some metrics I looked at, uh it looks already better in the last seven days, but uh the data is still a bit uh strange I think cuz not all nodes are updated with the bug

fixes, I think. Thank you, Nico. Do you have a do you have a link uh to the results just just as a refresher? Um Luca shared something in the chat above, but this is from a previous analysis. Um I'm really just mostly looking at attestation performance from load star nodes. Um but yeah, this metric might be a bit misleading as well because we uh basically put our RC on all our nodes. So that could be a factor as well. did some improvements there.

But yeah, I think it makes sense or I guess the question is is it urgent to do another analysis or are we okay and just uh go with it and just say yeah wait for the stable release and then confirm that it's a lot better I guess but then it would be really close to mainet already. So yeah, I guess this is a question for other CLS. Uh how confident do you guys feel about the current status of attestation performance? Any other clients comments? Yeah, we we're not we're not planning at this point to to release a new version

for the minute release. I think it's good enough the current and it may be too risky for us to even try to make a release with this old changes that we are still um we are still analyzing. So it will be something for for future release for sure at this point. Got it. Thanks. uh turns your your thumbs up. I guess it means to that you guys are feeling okay with it too.

We're okay with it. Great. Any other teams that want to chime in with their current situation? All right. Um I guess um uh we can move on. Um um mention I've put a message in the um chat. It we now have a bot in in the Discord ERD chat that is able to build

client images for you. Um so this is very similar to the tool we had at the interop event. Um, if you guys want other triggers, actions, etc. that you want to perform without having to wait some wait for someone to do it for you, then let us know and we can help automate it. That is very cool. Um yeah, any comments on this? Um yeah, from the Discord, uh it um appears

that part of us mentions that there's no builder building arguments yet, but I'm guessing that at some point. Um yeah. Okay, cool. Uh moving on. Is there anything else um from Peterra? Any any any comment from anyone concerns or comments? Yeah. All right. Uh looking good. That's that's great. Um yeah I think we can move on. Um

pure das um does anybody have the current status of peer das regarding devnets? Yep. Uh I can give a quick update. So Devet 5 is uh going down today, end of today I'll have announced it last week already that we would like to shut it off and if there's any if anyone wants to keep it around for longer than end of day today then please let us know otherwise we going to be shutting it down. Period number six has been around 76 75 to 76% uh participation most of the nodes are shine. Uh there's a syncing bug that they have. Uh currently

it's working in progress as far as I know. Maybe egg is here to give us an update. Uh there was also some issues with trending and refund refouse, but I've notified everyone about it and they're probably looking into it. Okay, thanks. Um, any comments on this opinions or concerns?

And from the chat, Roman says that uh they are seeing degraded uh blog processing times potentially due to the machines being used and they are currently looking into it. Yeah. So like going forward we might be opting in for a bit beefier machines because we noticed that uh pure does require a bit more compute, a bit more RAM usage. So yeah, just been trying to keep cost low as possible. I guess there's no way forward. Uh for Fusa uh zero, we're going to be using 16 gigs and eight core machines.

And Barto mentions that they were using 8 gigs until now. All right. Any other comments from clients um regarding this regard the higher resource usage for pure dust? By the way, is this expected? yeah, there's a lot more computation that happens as well as um we're testing with a higher blob count than before.

So, there's just more things that need to be stored in the RAM. And yeah, um I think it's relatively expected that 8 gig machines won't be able to handle it. I don't think I mean they can't really handle mainet anyway. Yeah, I mean to be fair the validators is significantly lower here. So the gig machines should still be okay. And ideally we want to keep testing with the gigs machines. But if we see too many nodes struggling with it, then we will have to bump it.

And Rafael mentions whether this is a problem. This resource problem is only happening on super nodes or is this a combination uh common problem regardless of whether it's a super node or not? Yep. It's right now it looks like it's mostly present on super nodes. All right. Um, yeah. Any comments regarding PRAS? Any specific issues that were found by clients uh during this that should be brought up or maybe just broadcasted?

Uh, one more thing. So, MV workflow should interior be working now also with Fulu. uh thanks to Justin's latest branches. Ideally, I should also be able to get that working quick assess by end of the week. Uh we have some small bugs to iron out and figure out what's up with the relay. Uh but once that's done, then should be all all good. Nice. Very good. Okay. Um I guess no more comments from for Peras. Um I think

we can move on or maybe one more thing regarding BPO. I'm not sure if you would want to bucket that into the same topic as Peras. Yeah, please go ahead. So uh last week Justin made a proposal uh regarding the BO uh work format. I hope every client team had a chance to take a look at that and give Can you share the uh can you share the link in the in the comment please? I'll share them one second. Do we have a open VR by the way?

I think it was just Discord link. Yeah. is saying that he thinks they should just be like normal hard forks on the I don't think the major question here is the side to be honest I agree I think the CL side's a bit more controversial Yeah.

Uh, I'm not sure the client teams have had a chance to really think about this. We might need to bring it up at the peer desk call more. Are there any comments from sales? Uh, or do you want to revisit it another call? Once they have uh enough time to to review this and re recommendations that he is going to review today about other clients um

if not we can just uh delegate it to another call. We are open to other ideas as well like there might be a better idea like configuration format wise and will mentions that there's going to be a discussion item uh for tomorrow in per meeting. Yep. Manu mentions also that they're reviewing prison. Yep.

Great. Um, all right. Since there not that much discussion to to happen today, uh, we can should be able to move on. all right. And the next item in the list is EUF. Um, so Dan, I think you were there. Um, yeah. All right. Um so we're getting close to um being able to unset EF DevNet zero and move on to DevNet one. Um over the last week I was able to get um a branch from Raitech of Aragon implementing EOF through both the transaction um the execution fuzzer and the validation fuzzer and they had a good couple clean

days to run on it. So we got we got those ironed out. So we got um six clients that run clean on um execution tests. Now um and that's for EOF0. For EOF DevNet one um we've got um we've started merging in tests in the EFT and we have only one more uh PR waiting and at that point u the main branch of EST will have all of the backwards incompatible changes committed. Um this last one has to do with the max uh stack size uh designation from a maximum stack size to a maximum stack increase. Um that is expected to merge here in the next day or two. Um I can give the PR a number of clients um are interested in

it. I think I mentioned it in the in the discord. Um but from this uh the other clients can start implementing and testing EOF uh DevNet v1 with their code and making sure it complies with the EEST tests. And from this I can get the fuzzers updated. Um, we'll need to add in a transaction fuzzer step as well because the TXreate um, one of the major ads for this version adds a new transaction type. Um, so I'll need to update the transaction fuzzer to start fuzzing the init code transaction types. Um, Basu and EVM1 already test cleanly with these tests that we have committed or about to commit in EFT. So there is validation and when we get more clients

we can start getting into the full fuzzing. Um there there's probably going to be a change in text crate coming up a little bit since we didn't get EIP 766 in um the notion of a pre-eploy probably going to be um not going to go over too well with clients. So um one thing we have right now is we have TX create exposed to legacy that probably will continue. Um we're also looking into other minor changes um such as either using uh ny method to deploy the pre-eploy um to a different address rather than the pre-ompile range or allow two equals nil um in the init code transaction and semantics on how that would work. So it work a lot more like existing create transactions. Um that's something we're

going to hammer out um on this Wednesday the UF call. Uh but in general it you know simplified a lot of that the deployment. We're removing the entire need to get an infrastructure to pre-eploy contracts which we were hoping to dovetail with 766 if that went through but since that's solidly out of Fusaka we are not going to be going that route with a pre-eploy in the uh the compile range that was the original spec design but we're going to go ahead and simplify that with that uh you have to devet one by Wednesday um no it's still targeting um turning it on about time uh time frame of uh petra are launching. So we still need to have at least a week or two for uh the other client teams who don't have DevNet one

implemented yet to de to implement them. I know REST's got a branch out there. I haven't looked into the RAVF stuff. I haven't tried that yet. uh the implementation spec is the um the PRs were updated and the DevNet one uh markdown page that um I don't have it on hand easily uh that uh was put up um not by Mario but by Perry I think um lists all the changes that we're doing.

Um let me find it. Yes, e pan ops devnet one. So that's listing all the changes and it's got the PR links to the PRs to the EIPs that are being updated. so that's that's a summary of the the deltas of the spec. The spec should be the EIP once those PRs merge. There's a comment on the chat whether this is uh just efing

right only ef a couple of things to e um and then we can make a release for the middle one and then we can uh test all the clients and then we can uh launch DevNet and I want to get some puzzing time as well before we launch the DevNet. There are a couple of tests that I want to write also for tx create because um we have different RLP tests for type 4

that we want to like generalize and bring up for newer test form test sorry transaction formats so I think I want to put some time into that uh it's I don't think it's blocking for the one uh we can we can defer it and generate it as a y yeah The only thing that might change is we might start allowing two equals nil. Um that'll be resolved by Wednesday. Okay, cool. Any comments from clients um regarding status of EOF implementation?

What's the status from everyone? for the UF implementation. Um, for us, we were waiting on the on the DevNet one spec. Um, it seems like that kind of stabilized. So, we will implement this uh in the coming week or two. Roman mentions in the chat that they

have a uh a branch for red. Any other comments from other teams? Um never mind. Visu argon ajs I have a branch for basic fully implementing it that's what we used to this that EVM mode is what we use to validate the tests and the other teams do do you guys have a timeline or what the status uh for Aragon I think it's a work in

progress uh with the tests and we have a different branch as well I think with more fixes uh other than the one that was mentioned by Dano or other than the racy tech branch uh Milan from our team is working on it urgent to join these calls next week and uh I think more and more fixes will uh keep happening but I'm not able to comment on uh the timeline yet. All right. Yeah, that's that's a good point. Um I think many deps are that are working on EF are not in this call. Um but yeah, if you can surface any

information in this core that would be great. So we can start coordinating on the other one uh release. All right. Um yeah, any other uh concerns or topics for EUF that anyone wants to raise or questions regarding the spec also I think that's pretty valid. Yeah, Johan mentions that they are also implemented uh the one spec.

All right. If there are not any other UF uh topics, um does anybody else have any other topic that they want to race that is uh not related to Pieras UF or maybe Petra going at one going at two going at three. All right, I think that's everything I got for today. Um, thanks everyone for joining. Um, there's a question from Barnabas uh about the branch for Eternis.

Yeah. All right. All right. Thanks everyone for joining. I think this is all. Um Um, thanks for joining. Thank you. Bye. Bye. Thanks.