All Core Devs - Consensus #96

Transcript

foreign [Music] [Music] foreign [Music]

foreign [Music] [Music] foreign

[Music] foreign [Music] foreign

[Music] [Music] thank you

foreign [Music] [Music] [Music] foreign

call Number 96 issue six two nine PM repo um the next couple of calls our canceled one is during Dev con and the other is a little break after Devcon um so this is primarily to recap um if anything needs to be discussed after the merge um and any of our couple other discussion points to kick it off the first item is the merge um I think by any standard the merge was successful congratulations

um are there any particular discussion points today about the merge about what happened or anything any issues that need to be resolved any unknowns that surfaced okay and I know there's um a couple teams on the execution there that have had some stuff Bubble Up And they're they're actually working on releases but I'm unaware of similar issues that have Bubble Up in the consensus layer so good work anything else on the merge

um mats do you want to take a minute or two to talk about like the base release I think you put one out yesterday or as if you can crack but yeah no sorry I was trying to get off mute there um no worries Yeah we actually we're actually we have a bunch of fixes that were ready for yesterday but we decided to actually delay 48 hours to probably be Friday at like midday eastern Time or U.S time zone uh the reason being is that we you know had even more fixes but we wanted to kind of test everything in combination so as not to upgrade users to ask users to upgrade twice within like the span of five days um but we yeah we we're putting out

something on Friday that will hopefully alleviate a lot of these uh issues uh that we identified that are kind of centered around the block import step um and that's what's causing some of the missed attestations that users are seeing and some of the other wonkiness with uh the consensus layer and then again with some of the things around empty block proposals we've identified a bunch of like cumulative fixes I want to say almost but we're waiting to make sure that we can test them in aggregate by Friday and then put something out on Friday so more news after that but that's kind of a rundown for right now foreign anything else on the merge um

Enrico we cannot hear you uh it's super super lag Mars do you want to go in the interim uh sure so we have this one it's not really an issue uh it's more like a quirk that gath is building a block when we get the um we get the signal from from from from the consensus layer and we don't update this block for until until the block is

returned from the consensus layer uh that behavior will change and what also changes is the additional 100 milliseconds that um we currently wait um if no block has been created yet so that means for consensus layer clients if they send us the factors updated with payload parameters and then immediately send the new uh the the

get payloads then we cannot then we will return an empty block all none of the clients should do this anymore um but it would be good to double check um because gas is not supporting that behavior anymore you mean that previously there was a timeout like 500 milliseconds that you that gath would use in this case to build a block um and return it back right but it will it will not be the case anymore yes exactly is there um an experimental build that clients could test if there's any

concern um yes I can link APR IL out of curious It's why have you decided to remove this timeout um it's just easier from this so we because we have this recommit now um that rebuilds a new payload every two or three seconds um and it's just not as convenient to have this this like listening on this channel anymore

all right thank you can you hear me now a little bit there yes yes okay cool um just to add about this what you said Mario so because there is there are some IED cases where we we do send for Choice update with attributes and then immediately get payload and there are the cases where um the beacon node doesn't know anything about the The Proposal so the API the evaluator clients calls against the beacon node to say here I am I'm a browser and I'm ready to this is my favorite recipient there are some cases

where this call never reached a bigger node and the value of their client immediately asks for for a block and in in this case at least in Taco we send the factors with attribute and then immediately get get payload yeah what do you mean by without attributes without attributes yeah for Choice with attributes and then it's very straightforward in sequence without any any any pose in this case pattern yeah do we know how often uh this case

appears like when whether your client is not able to send uh couple of times it's naturally can happen if you at least for the for Techo if you start bigger node at the moment because we don't persist that information on on on file system so if you're a started bigger node there is a little bit of time where the begin value their client uh way you have to wait for the beginning of the next Epoch to receive the new uh the new preparing information so if in the meantime the the valuator client needs to propose

there are some some seconds there that you can get a that behavior so we fall back on a default fee recipient but also aside the effect is that we do for Choice update with attribute and immediately get payload turns yeah so um another behavior that I think I wish you eventually address which I don't know if we have an issue today is that we do want to support some sort of like concurrent um payload building so for example right currently at prism at Slot 4 we don't see a block so we send the prepare payload with with the current head which

is probably the previous slot and then at slot five like like like a new blog comes in and then it basically becomes head and then we send a prepare pillow again but then at slot a we process all the aggregated associations and then and then they had uh basically reverse back so you kind of have this back and forth so I think it's like really useful to just like basically to support like a concurrent payload so therefore we don't have to make like bets just to make bets on like what we think the head will be how um how long before you actually want the payload do you do you do this so I would suspect thank you you would

give us the parameters right before so we only maintain um the payload for 12 plus seconds yeah I think Terence means in the interim of the slot if the head were to change based off new attestation you know it might be worthwhile to support concurrent building I do think that most execution layers probably this would be a very very very fundamental and deep change due to the way that um you know there is only one head at a given time and it's probably an intensive process to build but Mars we actually support both hot damn uh so so we we support

concurrent uh payload okay that's awesome so say if I so if I call a new prepaid pillow that is on a conflicting head uh you will now cancel the previous one right until the 12 second timer okay that's good so I probably will want to check like never mind and other Els Behavior here hey Myers so if I send two prepare payload messages with slightly different heads you'll be able if and then in four seconds I call get on both of them I would have results for both uh yes oh nice uh yeah I was on the impression that it

is not supported like we had this conversation before yeah and we can't do concurrent payload building it might actually not really work because if you because with the second one you will also send a fructose up to date and that will that will stop the so every new head will stop the payload creation so it's actually not working so okay that's what I thought got it okay so I think I'm going to create an issue and we can follow up with like discussion there

foreign Chris because we are at the topic of triggering block building I was thinking whether it's a good point to just add the subscription request that we've made yeah I agree why don't you give us a rundown I'll share the issue here yeah so the question is uh how can block builders get a trigger for starting to build blocks currently be using and all the builders are using a a fork of a big picture note that sends a a trigger tailored event or with the slot block hash and the

preferent now and this happens as soon as a new payload is available or after a timeout if a slot is missed and typically this is about 12 seconds before the proposer asks for for the payload for the bid and this gives the builders 12 seconds of time to build and to keep improving their bits and is there um if the head changes mid slot it is a new event fire in your build I am not sure okay it's a good question I think it should and I I know that our Builders they would support concurrent building and we would like to do the current building

and it would be ideal if this would just be an SSE subscription that's available on all the beacon notes it doesn't seem too hard to imp Implement and then there will be no need for any custom Forks and I wanted to throw this out and see if there's any any thoughts or opinions on that here so do you see the the trigger is new head or some time out into the slot if no new head is that kind of yes okay got it uh yeah any input the reference

implementation is linked in the issue so you see this as critical to not have a monoculture on the beacon nodes that um Builders are running yes because now every Builder on all the relays seem to be using the prison Fork here that has this functionality edit I think and this is only because a lot of shops are using go anyway um so ideally there would be no need for any custom fork and it's not a lot of functionality so it would really be ideal if it's just an SSC subscription and if a consensus layer teams would be on board here this would be I think a

really good addition to the future of the Builder ecosystem yeah any Dynamic input from consensus load apps today or do you want to take it to the issue I uh I think the Lord star should support providing the SSC events for the change head and with regard to how we process uh fcus uh basically we only send that to you before the slot starts we so it's really it is based upon a period uh like three four seconds before the proposing slot and if there is a head

change then at the time uh of the proposing slot we we basically send a new FCU as and basically wait for 500 the milliseconds uh on our own side before asking for a payload so this is the way how we process uh the yards and uh New Impact scenarios yeah I think the issue here is that the head event that is currently the city subscription is just coming uh way too late and we want to trigger block building like a long time before that as soon as possible which is like the new payload event the internal one and the alternative would be to pull

and manage timing and stuff on your own if you didn't have this SOC uh I think that would just keep continuing the current implementation of the the beacon mode triggering the block Builder with the custom API call as soon as a new payload event is seen internally I currently our Beacon note is triggering the work on the builders with an API call right so this is like an Express Fortress updated timing but yes like

it's also on for every single slot rather than for for validators yes exactly just a quick idea you can have an Interceptor between uh in between the CDL and the L and make the SSE part uh in this kind of you know middle middleware I think we've avoided putting um this software between execution layer extensors layer such that a failure in this piece of software does not break the node combination

um so I would I would argue against that yeah but yeah for Block building I don't know yeah um but Fork Choice updated is still not sufficient yes because it doesn't include all the attributes I think we need well in the timing you have to also have your own timing like the Builder would also have to say okay I'm partially through the slot I'm going to use my previous forecast updated um instead of waiting on a new head which I guess is logic that can be a bit of complexity yeah I guess this is mainly because we

can have like at least two competing forks and we don't know like Builder and relaya doesn't you know didn't don't know uh which one will be requested by by the proposer so if in the block Builder you view payload a is the head it should also build on on top of payload B uh if it is observed as like a kind of alternative and potential ad because in the view of proposer payload B maybe um the head currently okay

um I know we're on moderate low attendance today uh let's take it to the issue and and discuss there yeah thank you Chris okay um anything else on the merge any insights issues discussion points great okay the next item would be oh sorry Danny yeah I was just going like a quick thing probably not related to the merge but according to

the beacon chain the block Explorer I can see some slight fluctuations in the participation rate do people have haven't has anyone investigated into this do we like kind of know the the root cause of these fluctuations because it's sometimes really at 99 sometimes it is a slightly below that number so um a drop in three percent usually represents some amount of failed coordination on the zeroth slot of the epoch and thus that zeroth slot not getting the the checkpoint correct um that would be my hunch

for some of this especially when you see from you know 99.3 and to 96.4 I mean you would see these pre-merch um but if there's an additional thing going on here would be interesting yeah that's probably I haven't closely looked at this Statistics before the marriage so that's why I think it's like related directly yeah or in so yeah I mean I I definitely noticed them before the merge happening on the order of some amount of epochs maybe like one in ten if the frequency is increased that also might be a um some sort of signal that the head is having a bit more trouble

being propagated on the zeroth slot got it thanks Danny I'll do a plug uh in an hour and a half we're having a Twitter space about the merge data challenge investigating this kind of stuff uh certainly very interesting for that okay great uh Chris you wanted to give us a map boost update on the first week yeah I just wanted to just go for the

first week and see uh show a little bit what happened and how it's going um at the March everything went fine all the real estate for the agreed 16 epochs before enabling sending out bids there were in total five parties running release flashpot blocks route block native Eden manifold um I think we lost too much or I lost you okay let me repeat that at the March there are five parties can anyone else hear anyone I'm hearing Chris well I can have two yeah I can also hear Chris

okay well they're not just continue [Music] um uh the day after the March last Friday we received reports of a bug in math boost then the blocks contained deposits we uh wrote a postmodern here it took a little bit because it was an extremely busy week after March all together we did Security reviews we had a lot of people participate this plate being in test Nets we delivered 50 000 blocks on the test match but people didn't catch this case um yeah the postmortem is here we made the release with the fixes in on the same

day a big thanks to the Taco team for sending over the reports and helping us investigate going for about the uh outlined the Improvement testing integration and deployment processes also in this post text and the Really monitoring efforts which we will continue to invest more resources now since we have a little bit more available will also help find uh problems click stop here let's are there any questions about this bug just to recap like we received three reports of missed

slots because of that behavior probably very few more but not a lot more because we tend to get reports really quickly um any questions about this this bug it was resolved in version 151 so everyone running this released last Friday uh is safe and they haven't received any any other reports about problems next um I wanted to give a quick shout out to that all the relays implemented the data apis

which give insights into which blocks were proposed and also which bids they are received by the builders but it's not because of the relay so other parties can use these apis to to get an insight into but they are the pits at certain times of a slot that they really had available and what were the blocks that were delivered this allows building really cool tools like like this one here macmost.org by community members otherwise I wanted to share a quick picture of the validated registrations that the flashcards really during the

past week um altogether a flashbots really is now seeing four internal Builders and seven external Builders submitting blocks and going forward we are planning to continue collaborating in the open and contributing to the ecosystem as much and wherever we can and doubling down on the efforts on the really monitoring next we would also like to extend an invitation to get together in Colombia and discuss priorities and things we can improve and want to collaborate on more

intensely and yeah before before before before closing I wanted I wanted I wanted I wanted I wanted to touch on the problem with the really yesterday that was also shouted up before that was a problem with the block slot relay not delivering payloads for the bits which means proposals that included the blocks route really and the blocks are really having the best bits would would send assigned Planet beacon block to the blogstruct really and not getting the payload for it leading to missed slots uh this was a problem for about five hours yesterday

we were alerting the block slot team about this as soon as we noticed that something was off I in our point of view the correct process here would be to immediately shut down the relay and stop sending bids and every Relay operator should have a circuit breaker or at least a very easy way to instantly disable sending outputs that as soon as problems occur you can instantly stop selling Beats it basically just means return two or four status code on the get header API that should be like a very easy switch and we also expect the relays to publish post-mortems on any such

incidents and we hope uh that we will see this from the Block throughout team as well the block slot team was saying that they are going to reimburse the validators for them slots on the value of the bits I wanted to mention that here as well and it seems that about 88 slots from their counting were affected by this bug I'm happy to take any questions about this as much as I may have additional information I'm not sure I do but I can confirm that this incident was isolated to the block slot really and that this has been resolved yesterday

after five hours the validator registration count and the image you shared is that um actual validators that are using the flashbot relay or is that just people who like filled out a form online like what is that now this is the validate registration apis with the signatures of the proposals uh cumulative since the beginning this does not necessarily mean active proposals we are not currently tracking ethnic proposals right too soon but this is like accumulative all the proposited error registered through the uh Builder API presumably this is anyone who is at any

time started a med boost and pointed it at the flashbots relay yeah but this is just anyone who has ever done that at least once okay we don't have data yet on what percentage of blocks are being built by each of the relays foreign there is some data on mathboost.org let's make sure this again so just I want to add on to Mika's question right because I think based on the spec we do send registration like every Epoch so you can also do some pruning for example if you validator has

to be responsive for the last like 10 Epoch you can just remove them as well so therefore the data will be more accurate yes absolutely there is ways like that for instance to get more insight about the number of active validators we just really haven't gotten around to that but this is something that we would like to to to work on next as well sorry about my audio issues I'm I think I'm back um any other questions for Chris okay great thank you

um on Research specs Etc I did want to comment that although what is going in Shanghai is not set in stone um certainly withdrawals in 4844 will be a part of this conversation so on the consensus layer side if you are intending to do a deep dive or review or add testing or anything in relation to these specs now is the time I think there'll probably be plenty of conversations at Devcon and then certainly our top priority as We Gather again after will be to uh sort out what goes into

that next upgrade um naming wise cappella was a placeholder if there is a desire to have a vote as we did in the past by all means and then there's also a bit of a call for unified naming which if someone wants to take the charge on that as a discussion point today by all means but um I don't have much to add there myself okay anything else on Research spec

domain okay any other discussion points or closing remarks before we close for the day okay great work um enjoy the break from these calls and see many of you in Bogota thank you thank you thanks everyone [Music] [Music]

foreign [Music] foreign [Music] [Music]

foreign [Music]