All Core Devs - Consensus #106

Transcript

[Music] foreign [Applause]

repo since their call number

Danny Ryan: 106 relatively light scheduled agenda obviously if you have points to bring up in different categories by all means uh we'll get started off with capella which I believe is just a little bit over one week from now what's the precise date

Tim Beiko: April 12th

Danny Ryan: uh less than a week

Tim Beiko: 22 27 UTC um I forget what the e-book number is

Danny Ryan: cool if you are listening and you run infrastructure or nodes or validators Now's the Time upgrade um Perry we had mainnet Shadow Fork

three this morning right any anything to report here

Parithosh Jayanthi: yeah it's been looking really great so far we had two builders on the network this time I haven't heard back from one but the other one is producing blocks um we are noticing a couple of unable to Wi-Fi signatures but still looking into why um besides that healthy block production all client combinations made it through and we use mainnet releases for everything we also tried a couple of the standard procedures like deposits were made a couple of withdrawals were done with zero x0 potentials a couple with zero X1

and yeah just playing around with that sequence and everything looks good so far

Danny Ryan: the signatures that you can't verify those are

blocked uh those are magical stuff my Boost signatures I think

Danny Ryan: okay interesting go anything else on this any questions for Perry okay thank you and anything else on capella before next week great uh good work everyone excited to

see it go through um the neb there are no scheduled discussion points for tonight this week

Marius van der Wijden: um something that I wanted to put on the agenda um I would like to modify it blob transaction transaction type from Xerox five to zero X3 um I think it was was chosen in that way because there were some different transaction Types on some layer twos or something I don't think we should create the

expectation that mainnet would care about transaction Types on other networks and um I would yeah I would like to to modify the transaction type

Danny Ryan: yeah I um on this I think it would be good to have like maybe a high bit set for non-maintenance so that there was like just a different space that people could use as a standard but yeah I see the point

Oscar

Ansgar Dietrichs: yeah I mean it was basically kind of echo similar similar sentiment I feel like this is something where we should probably sooner than later start some

sort of standardization procedure I mean ideally there should also be like non-conflicting students between letters but maybe we don't care so much but um yeah it does seem like a bad president said to just skip versions on mainnet because I mean we can't really keep doing that so but but still we should we should look into standardizing this

Danny Ryan: yeah there is a little bit of precedent here at least in the consensus layer um I just shared a link to domain types from phase zero there is domain application mask which essentially just says if you're going to use the application layer and you're going to utilize the same signing scheme set this bit and you won't run into I mean that conflicts

um so you know does that surfaces in the AP ERC I'm not exactly sure but I think it'd be nice just to send it to check

Ansgar Dietrichs: do we have specific feedback for any either letters or Wallets on whether this would cause massive problems or anything in this specific case

Marius van der Wijden: um I have no feedback but I'm I also don't know why this was Children's Way exactly what it was chosen this way I think it was it was some code in argon so maybe the airplane can come

Proto

Protolambda: yeah just the color to any all two chain similar to the 10 IDs that we are pretty much that are pretty much standardized we should do A Better Effort it's standardizing transaction types and domains and whatnot um I think optimism uses one transaction type soon and so those arbitrum or at least I think they use up to seven transaction tabs I have to check and yes this current also

pre-composed

Danny Ryan: right is this this something Mario's does have a eipbr as well

Marius van der Wijden: oh yes it has a consensus spec PR and an

epr

Danny Ryan: yeah I mean I I agree with not trying to deal with conflicts because we're not going to be able to resolve it and then setting the precedent here that we just do sequential and maintenance makes sense um maybe it's merge these PRS shout into the void and see if it's going to cause a deeper issue than we expect

rather than I think merging the TR sends a stronger signal than trying to just knock on some drawers and see what's up especially considering we still have some lead time

models

Marius van der Wijden: yeah I think the only thing about merging these chairs is the question uh like what's the status of the test Nets is is it trying to just merge it right now or should we wait for another round of like I don't know changes

but

Danny Ryan: yeah I mean only because this layer side we do we'd have to do a release and pretty much the uh testnet would Target either the

current release which has gx5 or the next release which has 0x3 as for the test Nets on the eipel side um you know I think it's more targeting a permit or targeting head so we just have to send the signal but it seems easy enough to change unless except for maybe Aragon or others that have other types of transactions that we can collect and we don't know the complexity there

Marius van der Wijden: so the way I understood it from Roberto is that even Aragon removed uh this code the room of the transaction

type three that they had in their code base so maybe it's not even an issue anymore

Danny Ryan: okay so there's like potentially a little bit of coordination effort to like Circle back with the uh that's not schedule and decide if these could go into next or the following but um seems like General agreement to

plant a flag here and merge these um anything else on this one thank you Mars other than that related discussions I apologize I was traveling on Monday I am curious if I'm curious just like General State of blob decoupling networking work um if that's

becoming relatively feature complete or if that's still like a very active r d for teams

Terence Tsao: update go ahead sorry okay sorry and um yeah I can give you a quick update on the play some side of things so um we were able to implement All the gossip Futures we have been working on blocks by range and blocks by route and we have successfully tested that yesterday so that seems to be working the only thing that we don't have is bad feeling but I don't think that's like a test net blocker that's

not because a few weeks if not months to implement backfill it so I think like prism can probably uh pair with any client right now to start testing that so that's where we are

uh in

Sean Anderson: Lighthouse right now we're working on like the single blob lookups and integrating that with like our reprocessing functionality and like generally in Block processing and see that's the big remaining feature for us in parallel we're trying to work on like uh just a testable version of Lighthouse battery um could run on a local test net so we

start like actually testing some stuff that's generally

around

Danny Ryan: cool was the complexity generally as expected or were there unknowns and continued worries here

Sean Anderson: I would say as expected so far or it's like a bit of a pain but it's not it's nothing unsolvable

cool sorry

Gajinder Singh: sort of ready to interrupt with any other client and ethereumgs on the Yale side is also sort of ready and uh what I'm currently changing in load star

is uh the signing uh communication between Beacon and the validator to change into block contents from the earlier independent blob and block signing so that should anyway be ready by end of this week but anyway it's not really a blocker to participate in a definite since it's internal between load star and valid data as of now

updates

Ansgar Dietrichs: from Tego we are still in developing uh phase we are concentrating in the mostly on the API side for signing plus the

the logic for for caching and looking up a missed gossip to reconstructed bundles of blobside and blocks while we are we are seeing We are following the head so so all the complexity of handling caching and pools of objects and then complete the blocking part um yeah we are we have still uh work to do so we are not ready for for an interrupt at the moment

Danny Ryan: thanks

uh regarding the Ned any other items to discuss easy enough actually uh there's something on my mind on capella uh with the historic Roots revamp we essentially have historic Roots list that starts at capella um we don't have it full and we were talking about uh once that it becomes

fixed essentially once you finalize the Chappelle update you could in a future Quark have a fixed list that you can insert and essentially utilize the new historic Roots mechanism fully from Genesis that that was mentioned as a potential future item that does seem like a pretty low hanging fruit is there an appetite for specifying that and getting that into denib it's a bit of a question for yasik as well because I think he was uh eager to do so um but is there any

gauge of since I'm in on that essentially like left as a to-do from the last upgrade um and wondering we should tackle that to do here okay I'll Circle back with yasik and maybe just put it up as a feature spec for discussion and then we can once it's complete decide where to insert it anything else on dinub

um as

Mikhail Kalinin: we are with some you know to do at least what didn't help I am recalling something that about Slash validators and the Gatsby can proposer and probably some other miscellaneous things that at some point in time we have decided probably to put into dinner I'm just wondering are we still going to do that at least for this get picket proposal thing

Danny Ryan: was get Beacon proposer essentially filters out slash validators in the

selection process so that you end up with yeah yeah more full selection of blocks um yeah I mean I think there's there's probably room for a couple of small non-cross layer items but they would probably need to be bubbled up very quickly um you know next next time active discussion decision soon

Mikhail Kalinin: yeah like um yeah just yeah probably it's not super critical thing right so we can do it after before after dinner so I'll just wondering if it makes sense to or concurrently or just you know make sense make more sense

you know to defer it to the next one That's my kind of question

Danny Ryan: yeah I mean my general answer is probably along with a major feature there are cleanups and minor items especially when they're not cross layer there's probably room for a couple um and that we can't always push them off or whenever we're gonna do them we should refine those specs and have the conversation and if not make a more stronger commitment to them being in the next work okay

um other discussion points

Linda neb

Danny Ryan: um on Research spec Etc I did toss up a PR from dapline uh this General design that he and I were discussing on validator index reuse I brought it up not as a necessary point to decide where it goes into a fork or not but uh to gauge General sentiment here um I've heard two arguments uh one is it is really nice to reuse indices and not

have big gaps or not have gaps form and from an engineering perspective but also that you know the counter argument is you can probably handle this with some engineering complexity and not necessarily put it into the spec um I I just wanted to get some eyes on this and get a general feeling on if this which is a relatively minor feature worthwhile from the perspective of Clan engineers and if you don't have um comments today I just want to Bubble

it up to potentially leave comments in the pr one nonetheless this will probably emerged soon it's under like underscore features so it's not slated for pork but it's at least we're good at Future complete for discussion no comments we're kind of getting to the end of the list of our agenda

does anyone have anything else to talk about

Chris Hager: I could give a quick update about what happened in if the math boosts or relayed earlier this week

Danny Ryan: yeah that'd be great

Chris Hager: all right so let me post the post modem here that we posted uh two days ago on Monday a proposal attacker uh tried to steal a block from it really to unbundle and then change the transaction and propose that to make that sort of money it took flashing into account obviously

and they managed to do this by sending an invalid beacon block with a zero state route and at this point the beacon the sales clients they used to broadcast before valid validate so they really sent this block to the beacon nodes but the bigger node uh found it to be invalid so the proposal could receive the response from the relay and didn't have to raise a second block because that was invalid and for some reason not propagated quickly enough because actually it should have been um in the end they managed to still a ton of money from temperature

Imports and then they turned off their validators and deposit as a response of that we changed two things on the relate um in coordination with the other release as well um there is a a get payload cut of time that is around like four seconds now that if proposers try to request lit into the slot the payload won't be returned and more importantly now the CL client that we use they use to validate before Broadcasting um yeah because standard CL behavior is broadcast and then validate but for the relays we really need to the client to validate the payload before Broadcasting

but this introduces additional latency of course about 300 to 700 milliseconds for the validation before it gets broadcast and this pushes the broadcast time of the block further out um on top of this

um validation including the execution layer

Chris Hager: I think the clients do that our really at least already validates the execution payloads when we receive the pillar submission I think clients also do that I'm not 100 sure but it's all together and that the okay

parent photos maybe you can chat in here

Terence Tsao: yeah so um since then we have provided a patch such that we don't have to do ER validation in particular the method that we are using to validate before broadcast is rather slow because it touches four choices statements to stay and the plot to the DB we don't need to do that so since then we have provided a patch that just does the L consistency rule validation only and that is going to be much faster I think um this is going to be tested right now and get installed before um before the relayer is within to um

just use that validation

Danny Ryan: okay yeah that's what I was gonna get at is that there's probably a lot you can carve out there to you need to do some validation only

Chris Hager: yeah yeah totally definitely want to have codes that is a performant as it can get here to reduce the latency before Broadcasting um but all things together that the proposal can request the payload a bit into the slot like this is like one second one and a half seconds sometimes even and then the validation delay before broadcasting it pushes the broadcast timing um kind of into the slot further and we're seeing now uh additional

amounts of unusual amounts of Orphan blocks uh right now it's about I think four an hour up from like I think 10 a day or so because multiple release have the issue that if the validation latency and other latencies and more aggressive orphan will be given by RCL clients too that yeah the blocks they're just editing out uh so later there are more that we're seeing a lot more orphaning right now

Danny Ryan: yeah question are we seeing are these because people are beginning to make their

own economics

Danny Ryan: sorry Chris did you say you'd be back in one

moment I apologize I missed them yeah I'm very sorry

Chris Hager: my kids just had an accident and started screaming

actually

Danny Ryan: got it are you back and available or are you taking a moment

yeah I would say I'm back

Danny Ryan: are these due to

proposers beginning the Meb Boost Dance late into the slot are these uh orphans happening in relation to people who are actually beginning the membrus dance at the start of the slot My worry is that uh people who have deviated from honest Behavior sending a block at the start of the slot into more rational Behavior waiting to try to get more muv are the ones that are being orphaned here and that such actors who are rational and modify their behavior will modify their behavior and make the calls earlier in

the slot um so I'm just wondering if this is like an issue for honest validators or for rational deviant validators because if it's the latter I wouldn't worry too much and they're going to change the behavior

Chris Hager: can you repeat the question real quick

Danny Ryan: yes um is this a problem for honest Auditors that are making the calls to my Boost at the start of the slot or is this a call for rational deviant validators that are calling you know nav boost at two

seconds into the slot to try to get more mvv if it's the I wouldn't necessarily tune I wouldn't necessarily worry too much because they will tune their behavior to call earlier but if it's the former I would worry because now essentially that's showing that there is a too much latency in the in the comms

Chris Hager: yeah we are still monitoring and investigating yeah we will follow up with more data I think even for uh like all their client or configured to start the flow at T equals zero but

there can be a number of reasons for additional latencies here just a bad network connection can make the get had a request slow and then already start late and then by Randomness the validation is a bit lower than usual right and suddenly things get pushed back really far out

yeah yeah I understand hotels right

Potuz: so perhaps they can say a little bit about this uh the thing with delays uh even before this incident uh we were tracking New Yorks because uh not only Lighthouse but now prison is also trying to reorganate blocks

um we've been mostly successful uh we've seen some problems on slot one so where when the second slot of the epoch tries to reorg the first block on the epoch because the first block was laid then uh prison has an issue in that particular situation which does happen a little bit often because this first a lot of people in the first block of ebook oftentimes is left after the relayers started posting this this patches we are seeing much more reords I mean it's clear now we have more more missing slots and uh I do suspect that unless this has changed the situation is going to get

worse and worse and the the delays they're talking about is over a second to two seconds so this is essentially like a coin flop on each block the relay you're sending whether or not it's going to be reordered or not uh I I I strongly advise to look in this patch that uh Terence say that they're soaking because the difference seems to be between one to two seconds currently on the validation to just 100 milliseconds afterwards and with 100 milliseconds I think we're absolutely safe

Danny Ryan: got it and Chris you're claiming for it is 600

milliseconds is that um that's just the very highly resource machines like when we say one to two seconds are we talking about two different types of uh machines here

yeah

Chris Hager: I think it depends on what type of validation but generally like our we're seeing like 400 to 600 milliseconds uh validation delay on the beacon note account do you have maybe a bit less even but if you can get that down to 100 that would be that would be amazing

Potuz: and you also need to to recall that I mean the the time that makes the cut for us to reorg is not the time that we receive

the block is the time when we put it in poor choice which happens after our own validation so and that's the validation on the computer of the validator is not might not be a large computer so even if you if you send the block at three seconds that takes over a second to execute that's that's a coin flow a

contos I

Danny Ryan: I do think providing more visibility into uh honest

late reorgs would be valuable because I do think that a number of entities might have begun trying to modify Behavior to to get and broadcast late and that this combined with modified relay behavior is maybe amplifying the problem and so these actors will probably figure out themselves but just making clear that essentially there's uh a modification that is utilized by the majority of the network uh I think would help

Chris Hager: yeah I totally agree this is definitely

something that is pretty interesting and important to Monitor and to keep an eye out and to think about possible ways to to mitigate them that have also as little as possible negative impact on on solar stickers by increasing just regular missed slots for instance we are with the other release I know ultrasound really is also very active on monitoring their numbers and their performance and their mid slot and everything and we are working together with the other really to improve the parameters and the situation over time to collect more data and keep an eye on it there will also be more data coming out from this

yeah and we are working on Android hardening the math boost flow overall and analyzing potential attack factors and that could be possible by the way one more thing here is that there it would be really interesting to get more security research on Ms boost really code base and there was an audit and there were a bunch of people looking at it but um yeah there is um maybe interest in having a more coordinated wider Park Bounty program that is not not solely funded by a flashbot but

um I'm more of an industry advising Maybe

yeah

Danny Ryan: on uh specification of the proposer boost rewards that's landed in a PR in limbo for quite a while but now I'm just something like 70 of the network will be utilizing it any opposition to shifting that into spec even if there is a note that it's optional Behavior

Marius van der Wijden: so I'm not opposed to shifting uh math

boost uh into the spec but I would like to make clear that this bug wasn't wasn't an issue with math boost but it was with a relayer and so kind of fits like I don't know flashbot or the relay service providers have had their users uh the builders and are kind of trying to pawn this on uh on everyone else so I would like to make sure um like I don't oppose uh doing math like putting the the the math boost software into into the into like the general back Bounty program but

I very much oppose uh painting this as something that was like uh an ethereum fault or something this was uh definitely an issue of of these service providers and not of like the the general ethereum ecosystem

Danny Ryan: yeah thank you um just to clarify I was asking there's a specification point in the fork Choice that's hanging out a PR that has not been merged for quite a while even though most uh now the majority of the network

is utilizing it so I I guess there's two discussion points there one is how to provide Med boost with a better bug Bounty um and then also secondary questions of relay bounties and other things like um and the other is the before Choice uh composer boost rewards into specification

Marius van der Wijden: uh yeah again same argument I think this

this part is like Memphis is part of the uh it's part of the core and it should be specified

um and and standardized inspect out we have a table map boost is somewhat packed with the pillow specs already um what what kind of like different spec would you be thinking about

Danny Ryan: yeah just to make it clear Mars the Builder Stacks are under the Slash ethereum um org and our well specified and well standardized and

those are the maybe poorly named as I look at it but those are the communications with how to deal with and talk to my Boost so um I think that is well specified at this point okay on ish on pull request 3034 which is the late reorg specified by Michael Sproul it again seems strange that is not part

of the spec at this point if that's the behavior um even if it is optional Behavior so let's do a pass on that and make consideration over the next week or two on if and how to finally put that into the spec um yeah does this is uh I think there's a number of active conversations around that boost around safe relay Behavior around equivocations and other things I don't think this is necessarily done

uh but it'd be good to get the optimization in that helps with latencies and the continued monitor um is there any any other discussion points related to this incident or Med position General

POTUS

Potuz: yeah so there was a question uh that perhaps it's good to gauge here uh one of the options was for clients to provide an endpoint that you would submit a blinded blog and it would just tell you if it's valid or not uh just by checking the consensus uh validity of the block

and this was our first reaction of something that we would want to to have because having the relayers forking CL clients and having a prison fork and a lighthouse Fork is not a good status tool and I myself am divided about this I'm not really sure if this is something that we want to have if it has to be specified or not I initially disagreed on having this thing specified uh Paul honor I don't know if Paul is here but Paul seem to agree that this should be something that we would want to have and I just wanted to have this discussion

yeah

Danny Ryan: I mean my intuition is that if we don't then there will be a subset of

clients that are used by relays instead of a diversity of clients that support the behavior and the behavior will not necessarily be standardized or will be de facto standardized um whereas the RPC endpoint would certainly standardize it so I it doesn't seem it seems like one of those things that if we don't do we end up in a like a worse spot and if we do then we're kind of like adding this very specific behavior for this use case but knowing which is a bit strange but knowing that uh it prevents the actualization of a bunch of strange

modified Behavior

Potuz: right so My worry is that like by adding things that make uh make the situation to continue on uh except like diverting us from realizing that we need PBS and protocol and facilitating the the continuation of the status quo is not something that I'm happy about but I also agree that like having relayers fork in prison and and trying trying on production testing on production the present patches or or Lighthouse patches is not it's not good either

Danny Ryan: I guess the counter argument that is

members exists and it's not going to go away tomorrow um so there are leaving the issues to just stand there and it leads the system in a more fragile place because that is the Practical nature of the system today and If This Were a matter of like okay we're gonna do and try and PBS in three months because we're not going to shore up the system then sure maybe but that's I don't think anyone's discussing timelines like that

the Band-Aids seem necessary

uh when

Sean Anderson: we need something like this blinded block consensus verification I'm fine

Potuz: if we had PBS this would be probably some something needed for PBS so for PBS probably we will need some sort of value validation of the blog which is pure pure consensus anyways so this endpoint would be needed

you

Danny Ryan: need the workflow you would necessarily be using the endpoint right because you'd get that message from Gossip and you'd be doing consensus things on it to decide if you need to do an attestation to that like sub component of the block uh so yes you need the workflow no I don't

think it would you need the RPC but even then that's a reasonable additional argument

Sean Anderson: yeah I don't know it's like I kind of like the idea of having a separate endpoint but it's also kind of for a pretty specialized purpose then for like everyone to have to support it

Ansgar Dietrichs: if I mean even even if all the clients implement it I don't know if there will be on any case I'm super specialized logic to Super optimize that in points because it seems that every millisecond

matters here so does this prevent in any case having specialized fork for super optimize that one

they are

Danny Ryan: hard to say um but it at least allows the default relay Behavior to pick up a client and just do it rather than having to Port something obviously hard to say with these highlights otherwise actors will ultimately do is anybody willing to specify the

endpoint that you are for further discussion

uh I

Sean Anderson: I can uh like in the like in the next couple of weeks probably but right now I feel like I got a lot of the network on my plate it almost seems like this should be a part of the Builder specs even though at that point it sort of becomes like the Builder specs are execution have consensus

specs because this is the functionality

presumably a relay would be calling yeah like

Sean Anderson: like a relay implements um the Builder specs to be like yeah I guess it's more of like an internal API for the relay

Danny Ryan: yeah I mean the I think the I think the consensus apis is the place because it's just it's a functionality that a particular user needs a particular use case needs you could

obviously in the Builder specs have some notes about things a relay could or should be doing in between uh places in the communications but I don't think that's where you'd actually specify the RPC

Chris Hager: I think we should maybe consider the conversation in next week again or so because there may be additional checks that we've wanted as part of this flow too

Danny Ryan: yeah that sounds good

Chris Hager: I just wanted to say thanks for

everybody involved in chiming in into the analysis into mitigation into helping charge the path forward and and keeping around the data it was a huge team effort and yeah also um thanks for keeping this room here and I apologize that I just had a little small children incident in between and got closely unfocused

Danny Ryan: okay thank you very much um we will bubble up at least that discussion point for

next week and can discuss any other updates that have come since then um let's see nothing else on the agenda today any other items people would like to discuss before we close

um I believe there are at least a couple watch parties or chappella everyone upgrade your nodes and happy fork thank you

oh

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.