Dencun Interop Testing #32

Transcript

seems like most of the client teams are represented unfortunately I don't see anybody who's going to be able to live stream this call so I'm going to turn the record button on and then we can get started I will not turn the record button on because I do not have permission to turn it on um let me message someone and see if we can get it recorded but we can get started if not I think it's being recorded actually it's us recording at Le oh yeah yeah

you're right you're right okay I'll just message to him later and we'll get the recording thanks for mentioning that let's go ahead and get started then uh okay great welcome everyone to the Dene inter testing call number 32 doesn't look like there is a whole lot of things on the agenda today maybe let's get started with an overview of what's been going on on devet 9 since we just launched that last week does anybody want to share sure I can I get started so9 launched on Friday uh my mistake I made

a a bit of an oopsy by launching it the 170 validators instead of the 1,300 that I anticipated so I made quickly some deposits and uh during the weekend all the deposits have reached the pon chain and now we have the intented uh start so we have a Lor distribution of 70% in gu and nethermind and uh let me get the actual split um I don't have it uh ready at hand but

uh yeah 70% is G and under mind and uh we have Aron and bzu in the low 20s or 18% or something like that and etherum JS and r uh in the remaining 4% or so and so far all the clients seem to be staying up to head and uh just earlier today we had some prism issues but they seem to have resolved themselves just by restart and currently the only uh pair that is unable to stay Sy is nimus and ethereum JS which seems to be a problem on the etherum JS side um um that's it basically

for maybe uh maybe someone from some Cent can comment on yeah that' be great Merck yeah quick question does anyone know something about bad blocks because I see bad blocks three bad blocks being produced not sure if anyone investigated that B you can give an update on that front we are in fact producing bad blocks with bad version hashes we're still investigating it it's weird because uh we're fine accepting them but um we're producing them incorrectly we haven't

seen that before it's some type of regression so we're going to look to make sure that the trusted setup is being used correctly okay thank you so all basically all three blocks uh were produced by bz am I correct here I'm based off some analysis that uh Marius did and posted in the Discord I have not um looked any further okay good to know um gender did you an update if you're speaking I can't hear

you he unmuted but now it say yep okay yeah so on etherum side most of the issues that are coming up is because of some syn related to our Beacon syn uh basically we we seem to be having some issue where we are resetting the chain to Genesis and uh so they are tions that we are trying to roll out and see what the issue is and sort of rectified but on nimus ethereum J PA I don't see that is the issue and that

does not seem to be ethereum J problem because it's not even uh I mean there is basically the Fetchers are not getting the data from the PE uh to basically comp the beacon sink I think that is a issue and I'll check what is happening over there maybe it is it doesn't have good set of peers on its ends and I'll coordinate with Barnabas on that you're saying this is on the load star side it doesn't have good peers or were you talking for East JS so this is Nimbus ethereum GS pair and I'm talking for eum

GS what data are you were you trying to get over peer-to-peer for ethereum JS so it's basically trying to Beacon sync the blocks from the range about uh 3,000 downwards through Genesis okay I see so yeah okay makes sense Justin did you still have your end up um yes but it's regarding the hive test so if that's going to be a later agenda item I can hold till that uh yeah I mean unless anybody has

anything else on devet 9 that they wanted to bring up I think we can talk about Hive a bit yeah one thing so uh we did on dev night 8 uh spam test in with um sending block tool and uh it broke a few notes so maybe we will do it again soon and we will see if uh if we see the same issues yes on on defet 8 it was sent 16,000 transactions with six blobs each

and uh you know were're struggling with it uh on devet 9 as far uh we did test we' sending uh 16,000 but one block transactions and there was totally no problem for for the whole network like no issues cool yeah we'll be good to run that online any last def n comments is anyone here from me

team I don't see anyone yeah this would be like someone from flash Bots it would be great if we could start on boarding flbs or movie Builders I'm de at nine yes okay do you know is Perry here I don't see him um Perry is on H day today and tomorrow okay okay uh I can poke them but well yeah I'm also in a group with them so I I could also reach out to them and see yeah that's probably that's probably the better bet then yeah if you

say something great let's move on um Justin you wanted to say something about the hyp test I sure do so we have two issues right now that we're um we kind of discuss these in Discord so I'm imagining they're being handled um the first one was I think mentioned on Thursday or so it was the fork Choice I'll just paste the name of the test here since that's kind of relevant Fork Chas updated V2 then

V3 um valid payload building requests that's the one where we are basically testing first block past Fork um transactions are coming in and we are disregarding them prior to the fork if they are blob transactions per spec and so I was hoping that that test could be adj adjusted so that the transactions required for the first block are not sent till the for has been passed um the second one um is a issue called invalid payload attributes missing Beacon route um the test is expecting an invalid response based on the payload attribute however the fork Choice update

is a sinking State and so that's being returned before an invalid I think that's a incorrect test but happy to debate it yeah for for the first one definitely we are going to fix it the issue is that yeah we are sending blob transaction before the fork and I think most of the clients have complained that they they don't want to accept them before the fork and I think we should adjust that on hive uh for the second one I have not uh taken a look I will uh just dig in to see if I can fix it yep thanks Mario that's it for us and another question for for you guys

is the um so this issue that you're seeing for the M blocks is there anything that we can do on hive to detect it because it seems like we should be able to do it but great question I am very surprised that this was a regression as well and I'm not quite sure where it's coming from yet that's why I'm sus but it's a configuration issue because we don't see this on hive when I write locally so looking at logs at the tech will reject those blocks because version hashes number of verion hashes are different from the number of blob hashes so I think like you can have uh

some tests on those that's helpful thank you yeah you can post the exact uh error that we print out if makes sense for you to [Music] build some tests over it yeah because as far as I remember so we what we do is verification of the of the version hashes everywhere in Hive so it's really weird that we are not catching this um but if you have some locks or anything that we can take a look at yeah yeah it's just the number of hashes version of hashes doesn't match the blob hashes so it's kind of

that we receive I haven't counted so you can pick this log here because there are two sets logged and I think like the version of the hash size is different by the blob's hash size probably Maria do you have test that verify if client will not produce um blocks with more blobs than needed or something like that maybe maybe bzu added too many blocks to block I don't know yeah we do have we definitely have test where we verify that the client does not accept um client um sorry blocks with higher

number of blobs but I I believe that we do check the number of version of hashes somewhere in the engine API so this should be the place where we verify that let me double check and I will get back to you if not I'm going to add a another verification for this yep I just wanted to say on this missing Beacon routs we're actually filling the non syncing version because we are first checking if the payload attributes are valid and then issuing the for Choice

updated so it just seems like these two are a little bit at odds um because in the syncing version it wants to you know update the fork choice and check if the params are valid and then i' I guess like revert to the for Choice updated if the params are invalid um yeah another inconsistency Mario I want to bring in is again in the validations of the data itself for example in some cases what is happening is that uh for uh in new payload itself for example the version has not correct and one expect

in invited over here over there and in FCU then one expect syncing that is Justified because I mean the block itself is considered not seen by the client but in some of the cases it expects invalid uh so there is this inconsistency which basically causes one set of the test to fail if you try to you know do it this way that okay uh if new payload in new payload the data is incorrect and not the block execution and then you sort of treat it as if you have not seen the block but uh then you have so so then in some of the cases one expects invalid and uh

I think that there should also be sinking over there so there is some inconsistency in again how this is being treated in different ways in few of the cases yes definitely so the the thing here is that um we we have a set of tests where we send something invalid in the block and and normally this invalid uh field is part of the block header but in this special case this is some this this invalid property is just a property of the new payload request itself so I think there's no way for you guys to cash somehow that you seen this invalid property and that is fine um I think if

everyone agrees we can just simply remove the forkes updated that follows this this this check and and not expect the client to keep this invalid version at some point or uh yeah I don't know if if I if I made myself clear we can just simply remove the FCU in this specific check if if that if everyone agrees I think the current set of checks is nice but what I'm saying is that there should be one consistency either it should the FK Choice should think that it will get a syncing response or it will get an invalid response so what is happening in some of the cases it is expected sying and some of the cases it's expecting invalid

so it has to be either way or completely removing it is also fine I mean yeah I think I I met I recently just made a modification just to account for these kinds of fields where we don't have something in the header that represents the invalid thing that we're verifying um maybe I made a mistake somewhere that we are not expecting sinking for that I I mean uh or or we are what I mean is that we can make so that this kind of fields we we don't check for for invalid we we can just have syncing as as as a valid response um yeah let let me double check which

cases are doing this and I can get back to you with the with uh with the proper response cool yeah let's uh send that information to Mario any other comments or questions on hive tests either engine tests or ppec tests one comment sor sorry go oh yeah go

ahead sorry something met that you already mentioned on the thread that uh for the invalid transaction since we tend to uh also verify them maybe the invalid response should also be uh as part of the accepted responses in the test for the additional static verification you do on transactions right Yeah so basically we anything that we can do without executing the block basically on the Assembly of the block so it I think transactions is one and if there is some other thing then I'll also

sort of escalate it yeah I mean that's I think that's a good idea but I guess this also a bit of a question I'm curious like what other El teams think Mario you think basically right now we have a couple things that we're specifically checking to see if whenever you receive new payload if these things are invalid but there are more things that we could be checking so right now we're making sure the the verse and hatches are matching and I know that there are some tests that make sure that um The Blob gas limit is correct but we can do more things we could make sure that as Kender is saying that the max nons of the transaction is

being respected we could check to see if the um uh the gas limit of individual transactions are respecting the block gas liit there are more things to do here and I think it might be good to give clients a little bit of flexibility in deciding whether or not to return invalid I don't know if anyone feels strongly against that position so maybe concretely there are these Hive test checking the blob gas limit right now it requires clients to return

invalid uh what I'm asking is to allow invalid or um syncing yeah I understand so the the reasoning behind that was that so to check the blob G use I think you only have to check just a minimum part of the um of the of the number of transactions and that's it so I guess it was an easy check for for for clients to do that was my reasoning and since thgs was doing it I thought it was logic logical to to do that as as part of the check but I'm not sure if other clients agree yeah I mean it makes sense it

definitely makes sense you know I made the bigger question is I don't know you know when these types of things will arise on an actual chain and how clients will respond in the different scenarios so like if we get this and we're sinking how long is it going to take before we get to that payload execute and tell this consensus client that it was wrong and right now those things aren't necessarily specked out and so we have like gender is sort of finding this issue um with the nonces there are like some things that aren't um you know explicitly spec out that we don't necess have test for and so I think it would just be good to either explicitly outline every static check we

want to do or to have a little bit of flexibility on allowing uh clients to fail early or fail later good gender yeah so currently in the invalid transaction lot of invalid transaction cases uh so hi is expecting to be sinking but what we are doing is we are throwing invalids uh you know right at the point of new payloads and that is sort of causing us to fail the test so basically high is being flexible right now but ethereum JS wants to be a little bit more conservative uh so yeah if that flexibility can be built in that would be really

nice I I think the safest thing to do could be um so if there is a check that can be done um beforehand we're going to give the flexibility to return either in valid or syncing but if there's something that it's impossible to check and the client is returning invalid that should be like a failure to me so uh for example returning I don't know uh gas used if you don't have the parent or something I'm I'm not sure if there's something that cannot be checked and declin is returning hi invalid that should be a failure but yeah if if there's something that can be checked and the the client is returning syncing that shouldn't be a failure it's okay to just proceed with

the normal uh part of the test is is okay yep that makes yeah makes sense to me as well because if you return invalid becomes unrecoverable for for tech at least um one thing on these tests I think if it turn sinking it might be good to try and build a payload on top of it just to make sure that that sinking doesn't resolve as becoming like the head of the chain definitely so that's the normal testing workflow so what happens when you return syncing is that okay it tries

to build on top of it to see if it's really sinking then it provides you with barent for you to be able to to reconstruct the chain and then it tries to proceed with the with the check and then you should definitely return invalid um so we can do that if if that's how the de clients implemented right now for for for those specific uh Fields yeah okay cool I think that resolves everybody's concerns on that topic um is there going to be an accompanying spec clarification to go with that

do that an offer to update the spec Justin ah I'm just kidding um let's talk later maybe yeah maybe fair enough I could look at a try and update this spe later too also um yeah I'm I'm willing to put up a shut up sorry to put you on the spot there uh let's let's uh talk about me so we don't have to think about that for a second uh yeah let's yeah Circle back to me T on devet 9 I understand somebody from flash Bots has joined Shana uh

yeah hey do you want to give us an update on where you guys are at with respect to getting building infrastructure set up around the dev Nets uh basically we just have all the docket images ready for M boost relay and the Builder uh and we just need to have like a e to uh tesler I think like for the Builder we don't have everything ready to support kind of uh blunders containing blob transactions but I think that's fine uh for de9 that's basically uh yeah like what we have right now like it's code the implement is like where you just need to test

it do you guys plan to run a relayer for de n uh I believe like like Patos uh will like help us with like running that really so mean I can probably also get it up and running but uh I was just curious if flbs had any intention on like trying to set up the infrastructure themselves too I think we're planning to set up our infrastructure for defit uh think we plant is uh only do our infrastructure for test Nets like on G and possibly for hsky also

yeah okay I personally think that's a bit too late but maybe other clent teams disagree with me on this one I think it would be very good if we would have me workflow tested properly before we Fork any of the test Nets which means also flash BS running infrastructure uh themselves instead of us setting it up so in case something breaks you guys can iterate much quicker is that a

possibility uh setting our own relay on dev Nets yeah having flash Bots run some of this infrastructure and monitoring it uh uh I think I need to like Yeah The Talk the team to see like well at I think we kind of like not yeah not like capacity to support this right now is it mostly just like a engineering capacity you guys are lacking to support this or are there other roadblocks that

aren't allowing you guys to run the infrastructure test the infrastructure on dev nuts um yeah mainly in engineering capacity probably devops uh capacity right oh yeah of course it's obvious but uh yeah sha is programmer and I'm sure flash Bots has any uh um okay ignore me I'm happy to try to set this up for

de9 um but yeah I I think it would be just better if flash Bots were to run it yeah I mean let's uh let's continue this conversation offline I do think that this is a bit of a reoccurring theme and the building infrastructure is extremely important for ethereum and it would be really good to test this stuff now during the dev Nets before we get to public test nuts so if we can find a way to help you guys get to where that's like more of the common path I think that's going to be a much more positive place for all sides of the equation

cool any final comments on building infrastructure mbv for devet 9 it sounds like Barnabas is going to help spin up that infrastructure and in the meantime we'll chat offline to see how to accelerate future defet Builders uh could I get a update on which C team is ready for uh running these Lighthouse is ready is ready should be ready ready anyone from

Nimbus okay so I guess the only questionable client is nimus but possibly they also ready okay uh I will try to set this up then tomorrow and uh get back to you guys thank you barnas okay with that that looks like everything that was on the agenda um Mario you had some updates did you want to go over those yeah I already gone over them no the first one was already covered but the second one is that we have a new site uh for uh the interupt tests so these are uh let me share the link first

so this one contains a first will only contain go theum against all of the seal clients and will it will run as set of uh interrupt tests so just like a sanity check just a test net um a syncing site and a builder suite and eventually I will add a reor uh Suite that will be running continuously so you can check the results here I think most of the tests are pass passing for um most of the clients but I will keep updating this if you want to check the the the list of tests and the description of it uh um this is currently just a PR but it will emerge to master um uh in in the following days

but you can check the the Rhythm here and please take a look uh if you think there are some uh checks missing or something please let me know uh in general what happens in each of these test cases is that we we instantiate u a couple of notes U with uh half of the validators each then we start either on capella or we start directly on on the on the net Genesis then we go through the fork if we started on capella we start um sending blob transactions to the to the network we check on each um on some of the test cases we check for final session and when we do that we do do check every single Bon block and we cross check between the the beon block

the the blops the number of blops included the version hashes included and we do check that on the execution uh payloads to see that everything is is is matching um what other checks we do um um basically that that's basically it but um um but but yeah this this verifications are made on every single uh test test test Suite that we run the syncing the Builder everything so for example in the Builder uh tests we run the entire workflow and we also do the verification uh that the blob uh the blops the version hashes included the Bitcoin chain match what the Builder did the Builder included and also what is included in the execution chain um so

yeah a lot of uh cross cross checks in general but if if there's anything missing uh also we're open to suggestions or more uh test cases that we can do um just please uh leave a comment either or or or maybe just think about it and we can revisit this on the next uh me meeting but also please just leave comments on the pr that's that would be helpful also this is the pr link and that's it wow awesome work on that simulator that's amazing

cool any other comments guys for Denon testing devet 9 oh yeah I had a I had a request for testing scenarios I guess um it's related to this issue um tldr is like the proposer can send multiple valid blobs for a given um blob index uh and so I think Lighthouse would have had an issue where we wouldn't have been able to import a second valid blob so someone could have

been able to like split the view of the networks so I was just hoping we could test this either on this DEET um or on whatever feature testet whatever because it's would have been ini in Lighthouse not sure how other clients handle it exactly we're working on a fix it should be merged soon but want to bring that up have any other teams notied this could you repeat the the

underlining issue I think I lost the beginning of the description um yeah so since there's no like flashing related to a blob message a proposer can send um multiple valid blobs for the same like related to the same block and at the same blob index um without any penalty apart from like maybe reducing their odds of getting the block accepted um so what was happening in Lighthouse was we would forget if I think we would just import the initial blob and then and we'd ignore all other

blobs over gossip but for example if like the network accepted a different valid blob we wouldn't have been able to import that valid blob so we wouldn't have been able to like continue to follow the chain oh I think we like you we we just accept the first that we saw yeah so that's the that's the gossip rule that's the gossip rule but you could potentially send different blobs different part of the network sure yeah we haven't actually tested anything like that but uh this should be work on Teo because we we should then look up by root once the block has been

confirmed to be imported I suspect that yeah our issue is like our caching was stopping us from importing route right so you get a a cach level in in between that prevents you to do the actual look up okay oh we don't have that so should be fine yeah I just think it'd be worth testing this is like potentially a consens SP type bug yeah sure are we suggesting to change the gossip rule or no I think the gossip rule is fine I think it's like team just have to make

sure they're able to recover via RPC yeah yeah uh I agree gossip is fine we just need to Ure that we can look up everything once the chain diverged from the thing we think is correct good chender yeah I also think that current gossip is fine because if the proposer is sort of spamming then his block is not likely to get included which is basically a dis incentive for the proposer itself and uh obviously if you

see so what we do in lar is that if we for example see the gossip for the next door next block either blob or block then we pull by root the parent so that basically makes us uh sing to the to the chain head so just to confirm right here to solve this issue what we're doing is that say today um we receive blobs we save them and then we catch it and then we receive um some duplication of blobs

that we ignore it but the block actually points to the duplication so in this case do you just manually request the blocks over over our our PC um I think I missed the beginning of the question but yeah I think we have to just recover the RPC okay and then this is just RPC by root right yeah right did you get your question question

answered Mario more or less um so the B so you get like a a bid signature on a blob that is not present on the block or yeah that that could be a way that this would happen so like maybe two blobs that sorry two blob side cars that are identical except they have different kzg commitments and different like blobs but the kcg is valid there and if you get both of those before you see the block you don't know which one is actually in the blocks you don't know which to

keep I see I see so you you know the proposer and you you you receive a valid signature from that proposer but you don't know which blops are actually included in Block right ex and so you accept them all oh I see yeah yeah well so how it works now is you just accept the first one you see but then it's possible the block the block comes later you have to find the other blob side car I think I think the easiest way I mean we can even do this on hive testing if we have have like an RPC endpoint that we can use to mimic this behavior um is is there anything that we can do because a hive at the

moment for example does not have like a way of doing like a a a a malicious uh seal client so everything that we inject from hive to the um to the to the seal clients we do it via RPC so is there any way that we can um inject uh we we have the we have the dev valuator key so we can just fake any signature whatever but the problem is delivering this uh this spamming blops uh to the clients is is this is this possible to be a RPC in some way no I think we need to like inject the

messages via gossip yeah yeah I need to I need to then what what do you guys use to inject in in your tests uh gossiping messages random G maybe running the same key to different clients uh without running any slasher so I'm just thinking out loud so you get multiple blocks and multiple blobs for that so you don't know which one will win uh yeah maybe could

work yeah but there's there's no way to I mean yeah this is way just to be just to On Demand Being able to inject something to the beert beer Network some message that Hive sanks for example um yeah I have to think about it thanks for bringing that up Sean sounds like yeah we can talk a bit more about that offline trying to figure out how to integrate a test into our T infrastructure to look into that but it would be good if we

could execute the attack on devet 9 after clients take a look at fixing okay any last comments or questions uh I would like to discuss transaction Replacements uh like the case with replacing blob transaction uh buy the transaction with a less number of blobs so for example at neet mind we are not allowing two blob transaction to be to be replaced by one blob transaction and I'm curious uh what is the approach of other Els it was discussed a few

weeks ago but we didn't have agreement and uh I wonder if we should agree on some approach and cover it by hi test or uh just keep it as imple implementational detail of every client I would have to look to see what the GU behavior is I'm actually not sure if yeah if we allow that or not I would generally prefer to keep the high test around the TX pool pretty abstract and not um put requirements like this on Goss Like transaction gossip unless

there's a pretty strong reason I don't know how their teams feel about that but it seems like a reasonable requirement in the TX pool um okay no

comments maybe we should post it on Discord see if we can get a response from Vu okay still 11 minutes any other things we need to chat about or are we coming to an end I'll take the silence as we're coming to an end so thanks a lot everyone um we'll see you again here in

two weeks on the consensus layer call on Thursday and on Discord in the meantime have a good rest of your day thanks bye thank you bye goodbye