All Core Devs - Consensus #119

Transcript

[Music] time J

[Applause] for

Danny Ryan: okay I think we're live great thanks uh welcome to award DS consensus layer this is call 119 this is issue 8 74 in the PM repo just shared the link um seems like we have a light agenda today no one added anything so I made sure to add a couple of things uh so on denb uh if there's any devet updates uh any test net updates or discussion these are kind of the the things on the critical path uh to continue moving forward on this stuff um actually

and and we might have quick discussion on timing uh with respect to foring public test Nets because uh that was discussed on acde last week and it' be good to see if there's any additional people here that want to Echo or answer some input um then we can talk about research spe Etc if there are items and then open discussion on

dev devb on denb uh are there any updates regarding Dev Nets

Barnabas Busa: I can give a quick recap of devet 9 so devet 9 is now running for six days uh we launched it last week Friday and we've been quite good at participating

we have 90% uh part participation rate now uh we found a bug this morning with uh R it was producing in blocks and uh bzu also has a issue that they're working on right now and there was some latency issues with nethermind uh regarding uh Max blob counts maybe these teams can give an update on bit more details

Marcin Sobczak: yes today I performed a spamming experiment like with sending large amount of six block transactions uh to

the theet 9 and in NE mind we had a back with block building like we were doing too many reads from from theb like unnecessary ones it's already fixed and right now we are building blocks perfectly fine uh and uh yes this experiment with spamming is uh not finished like it's about five SK transactions to be sent more so we will see if there will be some more issues

Marius van der Wijden: um so I um I also did some like

manual transaction work for all of the eips so I sent a some transactions to called the pre-compile some transactions for the blob P up code 1155 5656 75 15 so all of the everything that is kind of within the ebm um so all of those transactions should be on uh defet 9 so yeah if you if you can sync def n then you should be able to like it kind of verifies that you have all of these features enabled

nice

anything else on Deet 9 yes uh M

Barnabas Busa: testing has begun so I have deployed now uh M flood which will uh spam some juicy transactions that could be uh basically harvested by MV boost and uh currently it's running on lar but we still don't really see any me transactions uh there's a relayer run by flash Bots now for de night uh here's the URL for it uh there's still some bugs here and there and hopefully we can resolve it by the end of this

week

Danny Ryan: great will we up the amount what what percentage of the network is actually connected to me boost right now

Barnabas Busa: 265 out of 1,325

Danny Ryan: cool

Barnabas Busa: about 20% um I also plan to update the El bad block generator um to um also produce use bad blocks on the execution layer with bad blobs

Danny Ryan: great anything else on dev9

is anyone from bezu here by chance

Danny Ryan: doesn't seem like it is

Marius van der Wijden: anyone from ethereum JS here

Danny Ryan: does not seem like it

Marius van der Wijden: so we there there's also some issue with uh ejs um sometimes encountering a bad BR in just being stuck on a bad Branch but I I think they said that they're working on it

Danny Ryan: got it Fabio

Fabio Di Fabio: yes talking for Basu uh since uh Justin that is following devet 9 is not here but yesterday he found a problem with the production of uh blocks that contain

BL transaction that after RoR so uh the problem should now be fixed and but I don't think is um is deployed in dep net so this is the the status update for

for B

Danny Ryan: thanks on the consens layer side anything of note um going on things stable cool um is there a intended timeline on

doing the limited run devet 10 to that'll test um 7514 the uh Max inbound activation VI

Parithosh Jayanthi: I think it would be nice to try and do that next week if we have all the fixes for devet 9 in by then we can also deploy the B Block generator by then

Barnabas Busa: I would like to see uh all clients being able to participate and no more bugs really found for uh 7 n before we proceed with the larger test so we should have a very stable like 100% participation before we move

on

Danny Ryan: are any of the issues and participation related to layer teams

you

Barnabas Busa: please repeat

Danny Ryan: are any of the uh issues in participation uh consensus layer issues at this point or is it execution layer

I'm fairly certain most of it is execution layer yeah

Danny Ryan: okay cool yeah I mean I guess if the purpose of DEET time was solely to test the the Q um then it'd be okay to move forward but I agree that just it tests the queue and does just you know an interesting load test so we might as well if we can have

everyone on both sides okay um anything else related to deits Mario

Mario Vega: so we added a single test case for the evil teu um scenario but we are also working on expanding this idea into more test cases um we're planning on writing a tool that will probably help us um reach more of the coverage and with this

uh with the timing of the blops but this is work in progress but just want to do that people know

Danny Ryan: cool yeah testing updates was the next thing um so thank you for the update and Terrence

Terence Tsao: yeah so um I know sham from Lighthouse brought this up have we started testing um equation block meaning that um a Blog to have multiple blobs on the same index they're equating but they're not slash but um we definitely have um seen some issues if that plays out that way so um we're working on some features to U mitigate this issue but you will be good to test it I can also help testing it but I

don't want to like step on someone's toe if people are doing that or already

Danny Ryan: so is this in relation to recovery when there was a bad one or is it's in relation to like dos or both

Terence Tsao: it's relation to Doss so say today you have a block and then it points the cas commitment to a block blob but someone send like a duplication of blobs with the same index but the kg commitment is different so the P2P validation filter will filter the second block and in that case you should use RPC request to request it but um I guess like some clients don't do that today

Danny Ryan: gotcha yeah I do think that um

some of the stuff Mario's attempting to do in Hive is directionally like this but more about um you know do you get the messages that you need rather than did you do the Dos mitigations that are going to make you safe um so there might be some complimentary work to do um in relation to to getting some of these messages on a devet point being is I wouldn't not do some testing here that you think is valuable um because I think Mario they're trying to capture some of it but I don't think they're going to capture all of it in

enrio

Enrico Del Fante: yeah I just want to uh say that

the a tle that was I implemented a couple of days ago was just in that direction trying to uh produce these heav equivocation blobs so the first block sent uh over the gossip was the one that is kind of malicious and then just after a half a second the tech web will publish all the blobs side cars and blogs that are correct so in case your client is not able to recover from that is because the first blob that have um a bad bad key um a bad bad proof in it uh kind of hide

the good one coming later and you're not able to recover even by RPC call just later so we fixed a thing on our site and um te now is able to request and recover from that but um yeah I don't know how easy is to actually Implement um and in include this a te in in a test net because I don't know if there are clients that suddenly um downcore the a for for for some reason and simply disconnect from it and since this is the will this will be the only one having the good blobs

you you're not able to recover in any case so yeah that's my

Danny Ryan: yeah so hopefully we can capture the like I see all these messages and I'm able to follow the fork um on hive but yeah I guess in terms of like spamming the network um the options would probably be to hardcode this as a a pier that can't be Downs scored which maybe uh I think some clients are able to do um but um maybe I don't know if that's valuable to put on devnet 9 or more in more of an isolated

environment

Enrico Del Fante: yeah I was thinking also expanding on that um um I think I realized that while we were doing the validation of the blobs uh we were not checking that the kg commitment inside the blob c car was actually matching the one inside the block so while doing the validation we were keeping the commitment from the Block and the and the poof and the blob from the blobs side car effectively ignoring what was inside the blobs side car so everything was going fine so we should maybe try

to uh build some fzy uh logic spamming with weird blobs that are to Capt to capture if clients are correctly correcting discarding these weird blobs because the gossip rules uh doesn't cover all the cases because the block is not is not available at that point

Danny Ryan: right Mario

Mario Vega: so yeah one nice thing about adding this to hi is that we can probably just keep a like a list of the every single B blob that we are sending

to the network and then we can just go through every single uh CL client connected to the to the testnet just to verify that we don't have the bad blob in in any of the respon from the for example for the from the Bon API I think that should suffice um to test what you're saying um the yeah the first thing is to get us a way to send actually send these bad blobs into the into the into the clients um once we are able to do that we can we can just simply make the the verifications of of the of the bad blops not being available anywhere in the

in the CLI

Enrico Del Fante: yeah sounds good to me because in in

our case I think we could have stored in in our database a blobs a blob side car with the wrong commit commitment at the end so by uh requesting via Bon API those blobs you should be able to to get those weird blobs from the DB

Danny Ryan: yeah I was just checking to make sure like blobs uh blob side car by rout you're requesting by the block rout so you wouldn't actually be able to respond to request unless you actually got the block and then verified that your blobs are correct in relation to it on the blob side car by range will

you respond to such a request if you haven't gotten the block yet to validate those blob side cars um I would imagine not but if you could then that might be like a weird accidental send invalid messages around path

Enrico Del Fante: at least Ino we don't serve any of those blobs that are completely uh in a separated area so it's kind of in a cach and not served at all

Danny Ryan: yeah I would I would assume that I'm I'm just going to scan this and make sure that we make a note that the availability

of the side car in relation to the block need to be satisfied but yeah I would imagine that just the engineering path through most clients would dictate that Sean

Sean Anderson: yeah so Michael from our team is also working on I guess similar to evil teu like an evil lious implementation um and the idea with it would be that it would like when it's proposing be able to create things that are like mostly valid but maybe invalid in these exact

ways we're talking about which would be like a blob whose kg kcg commitments don't match blocks or whatever um so yeah just just letting people know work on a similar sort of tool that we could maybe use as um a node that's proposing so if we deployed it like to a decent amount of the validator set on like a Dev net or attack net or whatever that could be pretty useful to like get to a lot of scenarios pretty quick where it's like partially valid um yeah

Danny Ryan: got

yeah just a quick I did look at the blobs by range spec and that it mentions that the side cars must come from their view of the current work choice and blobs don't enter into the fork Choice unless they've been matched up with a block so I it's good okay um anything else regarding testing Mario

Mario Vega: yeah I think I I will just

share the the link to the document that I'm preparing for for the blops here too so everyone can take a look um so the idea is just to keep on adding tests to this document so please just jump in uh read what we have written and if you have more suggestions uh please just comment on it uh so we can consider them if we end up adding this to Hive when we end up adding this to Hive um but yeah the main ingredient of those test is basically just having a tool which capacity is to uh interject um blops and then just send that blops to the network so we're working on that we have

possible solution but nothing is definitive but yeah in the meantime please just add your comments if if you have more ideas

Danny Ryan: thanks Mario can you drop that in um I guess the award devs or the consensus layer channel on the Discord with a quick comment just so that people have it out of this transient chat

Mario Vega: yeah of course

Danny Ryan: thanks Tim so last week we did you were talking about

tentative uh timelines with respect to beginning to for public test Nets um I believe there was there were a number of Slayer devs there um do you want to give a quick recap of that just in case there are additional people in this

Co

Tim Beiko: yeah so I think uh last week it seemed like we wanted to do devet 10 and then move to forking gorle and ideally do that uh before Dev connect um so that we can have at least one test net and potentially some time seeing it live before it happens um so I guess if you know based on on what Barnabas was saying a bit earlier around wanting to see the clients run clean on devet 9 and then launch devet 10 um I don't know how realistic it is to do all of that next

um if we could get it sorry if we could get it done before allord devs I mean we could then figure out when we want to Fork Gordy on allord devs um and then I think pretty much if we can't do that then the next CL call so on the 19th is when we really need to uh to to have done it if we want this to happen before death connect because you know we if we picked a Time on the next uh award devs then it could happen you know sometime late October like uh you know two weeks or so after that and then the sale call after that starts pushing into like early November and uh we

probably want to get this done before people start heading out for def connect um so I guess I don't know like how the people feel about potentially uh being ready to move to Gordy and say like a week or two and I think a question as well is like how um how close to like having everything on master and put you know packageable all in a release our different clients um because ideally if we pick a date you know say on the Thursday then we can have the blog post out on like the Monday or Tuesday so like you know give a couple days to clients to put out of release but it would be um it would be ideal for that that like stuff is

already in master and it's more a question of what we put out then trying to bring a bunch of separate branches back into master and potentially having to spend a lot of time on that um yeah

Danny Ryan: Andrew

Andrew Ashikhmin: um yeah in terms of Aragon Readiness uh everything is merged into our development Branch so we could release reasonably quickly uh I have a question about the key kzg ceremony is it is it complete for production and what do we want to have our proper final kg set up for devet 10 if not at least I suggest to do it for

girly

Danny Ryan: yeah I think we certainly wanted on Gorly I does anybody Carl's not on the call does anybody have a view into where we're out on that

yeah I know Carl's working on it he had hoped to get it into the previous devet but had some issues that needed to be resolved and is still working on it with the intent of getting it into devet 10 I believe I think maybe Alex was also working on some of it or or helping him so if there's anything else Alex please add

me Alex

that sounds like you were not involved yeah thought you

no I was yeah well very tendentially um but yeah as

far as I know that's on the way I thought that he had sorted it out but is he on the call I guess not uh yeah I can follow up even later today and see what's going on there

Danny Ryan: yeah I just pinged him too Terrence

Terence Tsao: yeah I think my naive intuition is that like I think we're a bit early in talking about like 14 test net or even a test net schedule like timeline aside I think like what I would want to see in the death net before I put confidence since working on test net is

that like we test more like relay infrastructure which is good today we have a relayer that set up 20% using the relayer but I would like to bump up the number and test that slightly longer and another thing I would like to see play out more on the death net it's just the blob scenario testing that Mario was sharing that if we can like run more test through that we'll have more confidence then again like clients still have bugs today so I think it's like personally take is that we're still too early talking about like

for testet

Danny Ryan: are these things that you if you saw over the next week week and a half that and things look stable then then

it's time to have that conversation or do you think that it's a Time Horizon that is longer than that in terms of like seeing spam instability around that

I think like

Terence Tsao: I think like one to two weeks maybe too soon my take is like two to three weeks each that we should see like Improvement but yeah I don't

Danny Ryan: because you suspect that there is going to be stuff that falls out of it you're see bugs in it okay so I guess we're in the phase where we need to be revisiting this on each of our calls to kind of get the updated status on testing and subsequent bugs um so that we don't let it slip but that uh

you have low confidence that especially next week that you're going to be ready to do so um and potentially in the two-e time Horizon we're beginning to get some clarity but no commitment yet

enrio

Enrico Del Fante: and just want to Echo what's what teren said and also add the fact that I don't think we have H tested any interrup between clients considering the fact that we have a new block V3 API and te is not has not completed the implementation of that yet so

this is Deon no Dev validator client interrupt

Enrico Del Fante: yeah so if there are out there guys that are having using using mixed deployment me to run girly and well whatever test that we decide to go to we should have all the use cases so um more test on the on the Builder flow Plus or all the API the CBN API tested has everyone implemented V3 I know a couple weeks ago that was not 100% Clarity on that Has any team not implemented the new methods Sean

Sean Anderson: oh yeah so for Lighthouse we have an implementation

that's unmerged it's like far along but it's not in yet and just to give like a broader Lighthouse status update we're in the process of getting didn't have merged into our unstable Branch but I think we still need like another week to get it in and then after that we'd like to have a decent amount of time before we actually cut a release to do regression testing on that because it's just like a massive set of changes so similar timeline with other teams out there

Danny Ryan: got it thanks um would it be valuable to do a bnbc interop related short-term devet once

people have all of the um all of those changes merged is that something we've done in the past how do we usually test that have we ever tested that

Mario Vega: so sorry

Danny Ryan: yeah go on

Mario Vega: um so Hive is using beon GL validator setup everywhere so is this something

else that we are talking about or

Danny Ryan: well it's it's mixing the two it's mixing to uh so you there's a there's a common API between the two so I could have teu be and light Health validator um and some people presumably do so on Main net at least

Parithosh Jayanthi: in the past we haven't really tested that on dev Nets um but we if there was an issue it usually spills out when we F the first test net because there's like yeah but there's also like a million different combinations you can run everything in so there's only so much you can test

Danny Ryan: well was a very finite amount of combinations

yeah you

adding well

Parithosh Jayanthi: yeah but now we also have the Builder API directly connecting to the um relays without M boost and we have third party softwares like vouch so there's kind of the Matrix keeps expanding

Danny Ryan: is it worthwhile at least doing some sanity tests around bnbc combos on hive or is that not a path we should open

Mario Vega: I think it's possible um right now it's hardcoded so if you start a teu beon no it will automatically start a t teu validator client but we can make

that configurable and I think should be easy just to do all the combinations but given that we are focusing on blobs right now I'm not sure if we can do this uh in the next couple of weeks um but yeah it's doable in Hive

Parithosh Jayanthi: um actually if we want we can just do this in defet 9 right now I don't think it would be overly difficult to change our anal setup we can just add a bunch of validators and spin up more instances um which clients claim to be ready so that we know who to test for

interrup

Sean Anderson: uh so all clients still support

the block V2 end point right with denb types DB types because we don't have the V3 endpoint in Lighthouse um our validators will only call it V2 though

has with three

Dankrad Feist: we have one and we have implementing

tree

Sean Anderson: okay well we'll work on getting V3 in

pretty soon so we can test interrupt with other um consensus teams

Danny Ryan: yeah and and Mario in terms of like the criticality of this getting into Hive I'd say agreed there are much higher and more important things to get in there I do think in the long run having a few sanity tests that do test these different configurations of bmvc um would be valuable I wouldn't I mean I don't know the load and run times on hive necessarily but like I wouldn't necessarily run them through like all the tests I would run them through like you know Canon out station be made can a block be made and like call it a day um

maybe a few others

Mario Vega: yeah there's one perfect test uh Su which is basically just the sanity just starting the N you start sting blops you see everything works just wait for final session and that's it so I think there should be that should be the ideal yeah okay ideal one um yeah the problem right now is that is very there are a lot of assumptions in the in the way that the the clients are spawn so there are a couple of important changes to be made if you want to do like the combinations but once we do that it should be easy enough to to prepare this this uh combination testing

Danny Ryan: yeah let's uh in terms of Hive let's put this on

ice and pick it up in a couple weeks and see if it's can be prioritized at that point okay any other um comments with respect to test net planning um in relation to testing in relation readiness

Etc

Zahary Karadjov: I would second the opinion that the Builder infrastructure is kind of the most lacking aspect right now

Danny Ryan: okay so we definitely want to wrap that up over the next week or so and to

begin to see um a lot of messages channeling through their anything else on this one anything else related to denb okay um any other discussion points for today regarding research

specification or anything else okay great uh thank you everyone give the good work um we'll sync many of us will sync in a week on the execution

l call bye take care bye by bye goodbye bye byi

Happ he

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.