All Core Devs - Consensus #92

Transcript

[Music] [Music] so

[Music] [Music] [Music]

[Music] [Music] so

[Music] [Music] [Music]

[Music] [Music] [Music] [Music]

[Music] [Music] [Music] [Music]

[Music] [Music] so

so [Music] [Music]

Tim Beiko: we are live

Danny Ryan: great welcome to consensual air call number 92 this is issue 574 and the pm repo if someone can share that link on youtube i don't have the chat handy thank you so we'll go over uh anything merge related then we'll have a segway into mev boost um some current points discussion any other client updates and general discussion from there give me one second and then we'll kick it off all right cool

uh before we get into any of the particular discussion points um perry is there anything to share with respect to any of the shadow forks any of the um testing going on on the side

Parithosh Jayanthi: yeah uh so we had two shadow foxes alaska we had a goalie shuttle flock that happened last week and one of the things so we didn't really find any big issues i think there was one that i don't recall right now but it should be fixed by now uh oh yeah though one another my team i think wrote a big post about in ether andy um yeah but besides that we've been testing mev boost on golly shadow fork five it looks good so far we started with about

eight percent of the network and i think now we're about 30 of the network um i looked at the orgs i looked at late blocks stuff like that everything looks normal from my side but in case someone notices something or has an idea on what else we need to look at then please do let us know uh we had mainland shadowfork yesterday that marched on tuesday and there weren't any outright clan compatibility issues so that's great i think it's the first time we didn't have any problems um but now that it's been a few days i think a few of the aircon nodes have lost peers slash are failing to find blocks but that's part of the aegon doesn't really like the shadowfox

um and i had to re-sync a couple of the bezel nodes because they were run on older version um and i think they're still not up to head yet but in general the network's looking good we have healthy proposals otherwise and we're good to go we would be having a curly shadow for six next week and the idea for that one is to have mev boost enabled before the transition so we can also test how the transition would look

like enabled harry for the uh for the macbook's testing or um which class are you testing me right now

Parithosh Jayanthi: uh i'm just going by the flashbots um

chart so it should be prism as well as load star and deku i think okay thank you so those are the three that are listed as yeah those are the three that are listed is finished the i think lighthouse as well as nimbles are both still pr's

Danny Ryan: can you share that chart in the chat

uh yeah of course cool

Danny Ryan: any other questions for perry or general comments around shadow forks or testing awesome uh great work on the last one

okay the next agenda item is the execution layer behavior around terminal blocks uh this has been a bit of a conversation over the past week on um what happens what to do around multiple valid terminal blocks and if the consensus layer picks something that is unexpected from the execution layer and how to handle reorgs and mikhail can you give us a tl dr and if there's any questions or any like words of advice uh make sure to pass that along

Mikhail Kalinin: yeah sure um yeah that's related to godish shadow fork basically the the case occurred uh there uh with the another mind in particular um yeah just let me give you a quick um

description of what has happened so there was like a terminal block a that mind received by gossip uh it's been imported um and uh yeah this will occurred terminal total difficulty and after that uh the other terminal block let's call it terminal block b has arrived via gossip but it another mind has this optimization it does not process blocks that are not becoming the head of the chain like similar to aragon behavior so the another mind put this uh terminal block b into their block tree but didn't process it so there is no state for uh there is no like post state for this block then um

uh the um the first the transition block built on top of terminal block b that hasn't been processed by by node has arrived via a new payload method call and what happened next is like that the mind returns thinking um because it does not have this uh does not have a state to uh validate this transition payload and then consensus layer has these safe slots to import optimistically um thing which protects us from poketrace poisoning and here we get the compound

effect of the two basically the node get got stuck because it can't import uh it can't switch to this work for 128 blocks uh the default uh value for save slots to import them into basically and that's it so the um the outcome from this uh case uh is as follows uh the expected tl behavior uh around the transition block is that whatever happens with like uh multiple terminal block received uh once uh el has received a transition block and one and when it has all the data required to recreate the current state and validate

this uh transition block it must do so so the fix is uh like uh i i i think in my opinion uh and we have been in touch with merrick uh from that mind discussing this uh the optimal fix would be if you receive like a transition block uh that is built on top of a terminal block b like in this situation then you have to execute terminal block b first and then execute a transition block and respond with the correspondent status like valid or invalid to allow cl to proceed with this with this branch that's actually it and this is what is

expected basically and i'm curious if i don't know [Music] i guess that uh go ethereum behavior around this is that they are processing all blocks uh that are received from gossip maris can you confirm that

Danny Ryan: all blocks up until a valid terminal block and then not necessarily the children

Mikhail Kalinin: yeah

but on any branch yes and we process all the sibling blocks of the termite block

Mikhail Kalinin: so they are instantly processed upon

receiver receiving them from gossip right

Marius van der Wijden: i'm pretty sure

okay yeah that

Mikhail Kalinin: yeah and i believe that aragon has a fix for this as well um i don't know andrew do you do you have any um any information about that

Andrew Ashikhmin: fix uh sorry regarding what like multiple terminal blocks

or what

Mikhail Kalinin: yeah like like uh aragon um yeah there was like a pr in

in the discord shared in the discord that actually do the execution memory or whatever uh on the

site work

Andrew Ashikhmin: uh yeah that's right that's still somewhat experimental and uh i also have to double check our logic um so specifically for multiple terminal pow blocks i i'll look into it

Mikhail Kalinin: cool and thank you um and i'm curious what is the dazzle behavior in this case

Gary Schulte: i don't know off the top of my head to

be honest i'd have to get back to you on that exactly how we we deal with uh conflicting blocks coming over gossip for ttd i think uh our expected behavior though is that we're going to not reorg unless we receive from the ceo an expectation like for choice update uh that will reorder to that different ttd block

Danny Ryan: right when you just improve work mode or improve stake mode when you have short range forks say two ancestors different um do you execute those or do you just accept them and then only execute on the four choice

overtaking like do you execute short range forks or not

Gary Schulte: i think that we're uh in proof of work mode we're going to just rely on the uh heaviest chain and then from then we're not going to reorg at all until specifically we get direction from the cl so a different a different block coming from uh gossip would probably probably reorganize ttd block but we would you know immediately switch back when the client consensus supply directs us to

yeah and

Mikhail Kalinin: it's important uh when user when you're switching back and you have like

this block not yet processed it's important to to have it protest this is like one of the ways to uh to to resolve this particular thing yeah why i'm asking is oh yeah merrick nice here

Marek Moraczyński: uh hello yes so uh i think uh gary you you can have the same issue like we had so we didn't process the block and we returned syncing so uh it will be good to verify it in the best zoo uh feel free to ask me if you want

to more details yes definitely

Danny Ryan: if you if you do execute these short

range forks then i think you're generally protected um but there's probably needs to be a no added to the spec that kind of all of these all valid ttd blocks up until uh probably finalization should just be executed immediately even if it is expensive and i think that would resolve the issue here and we also need to surface a couple of tests here the interesting thing here was that ttd terminal block b was of less difficulty than terminal block a and so it wasn't like by default or executed and i think we can easily get that to a test as well

yeah

Marek Moraczyński: i think one of the other considerations

is as soon as we hit ttd invasive we immediately disable uh our block propagation manager so we probably have to be a very we have to have to be a tight race to to get an alternative ttd block we're gonna we rely on a backwards sync in that case to reorg back to a different ttd block

gary

Justin Florentine: i think we wait until the second finalized block before we disable that propagation

yeah you disable probably propagation of well i would hope disabled appropriation of children of and and descendants of a valid tt block but um i think the spec says not to turn off the gossip until something's been finalized

till it's finalized

Mikhail Kalinin: yeah yeah this is like important uh part of of the transition to have the um the ability to uh to switch between terminal blocks and yeah we will have a uh yeah mario can confirm we'll have a test that reproduces the same um scenario in hive

yeah sorry go on yeah

Mario Vega: that's correct we are working on i uh this is a specific test case um to reproduce the the gossip issue one thing i would like to mention is it would be favorable to have the specific

behavior expected in the spec for example how many blocks do we expect to gossip after the ttd and so on and so forth i'm not i'm not sure it's already included but it would be nice to have

Danny Ryan: yeah michael and i to make sure that this edge case is very clear

Marek Moraczyński: two additional things from my side that we should be careful uh is that in new block message we can send we are receiving uh terminal uh not terminal total difficulty

but we can't rely on that so please be careful when you are doing checks uh and what is more uh we need to reject everything after terminal blocks from proof of workchain otherwise we we could have problems

right

Danny Ryan: so all chains should be process incorporated and accepted via gossip up to valid terminal blocks and then no children no descendants of such valid blocks and then after the chain has been

finalized you can stop gossip on that layer okay but yeah we'll we'll make sure that this is abundantly clear in specs and make sure that a test gets out soon anything else on this okay next up um exchange configuration before ttd is actually set for mainnet um there's a desire to do this to allow people to get their setups going um and probably some details to decide if we're gonna do that paul can you give us what's going on here

Paul Hauner: yeah sure thing so previously we've talked about enabling this endpoint before the ttd is set so the purpose that we want to do this is so that people can set up their um bigger node and so their execution layer and consensus layer in a one-to-one relationship like they would before the merge so i think all things turned it on which is awesome however it seems that from the eel side it's unspecified how they should like what the values they should put in the exchange transition configuration struct when the tecd is not announced um the consensus layer has um a standard around this we have these two the two to the two five six minus two to the ten values that i believe all the consensus clients would use um so

one option is for the execution clients to just use those values um and then another one is to kind of modify what we have now to send nils or nuns around um i have a slight preference for the first one um it means i don't need to change anything as a disclaimer um but it's also kind of um an existing specification that exists that could be expanded um and then everyone would work with it but once again don't feel super strongly about how it goes

Danny Ryan: right so that value was um put in to our main net configurations for the ability to do testing and to do so without with the value that was sane i mean a

value that was not going to accidentally uh trigger this on main net if things work their way into mainnet um so i do think it's a reasonable value to show up in there if people want to be able to do the configuration

now yeah

Paul Hauner: so i think it would probably mean for all the execution um clients to just have this kind of one really big number that's a constant and then instead of returning um like nil you just return that constant and check against that constant

Gary Schulte: what was that constant again is 2 to the 256 uh could you repeat that again

2

Paul Hauner: to the 256 minus two to the ten um in that merge interrupt channel i've linked it a few times also on the issue for this one yeah danny's got it too it's a weird number

Danny Ryan: yeah the history on that number is because we do use it for test vector generation we didn't want to go over to the 256 by accident by creating some blocks on top of stuff [Music] so almost the biggest number okay

any other comments on this any desire not to do this okay i'm not exactly sure if or where this number should land other than in code bases i mean it's in the consensus layer specs as a placeholder until the final releases but i don't know if there's an equivalent place that should go on the execution

there

Marius van der Wijden: so the problem is if we start adding it to gaf then we will have like a big like messages all around the place that the merge is configured at this point and then there's like this really big number

yeah

Danny Ryan: if not configured use this value for exchange configuration rather than nil i think that would probably be the extent of it yeah

yeah that

Paul Hauner: yeah that'd be cool we don't need it to actually actually be in effect just lie to us

Danny Ryan: okay if someone from the execution layer thinks this should land in a spec or an issue or something to help coordinate then by all means do so um but i don't know where that place would be

Mikhail Kalinin: uh yeah my preference not to include it

into the stack yeah basically because it's just a like one-time functionality i guess it's just easier to have it in code basis and that was like one time and done for

Danny Ryan: okay i'm fine with that okay mikhail you have an issue open on the consensus layer specs um i don't know if it's i wrote clarification i think it's an extension can you let us know what's going on here

Mikhail Kalinin: yeah there is a pr um that extends the optimistic the definition of what is an optimistic node and there's small context behind this

um yeah suppose like yeah we have like currently we have an optimistic node definition um as follows another is optimistic when it's head is an optimistic block when his head is optimistic um and the case uh um the case that these pr addresses uh may happen due to optimistic sync but it's not like uh related to optimistic hat directly uh so the case is as follows just imagine you have like a you have a optimistic branch branch of optimistic

blocks that justifies some checkpoint so it has just it has enough of the stations to justify um the new checkpoint and it goes beyond the ipob boundary so the checkpoint is actually becomes justified and this justification comes into the block store so the justified checkpoint is updated is increased um in the block store but but it happens uh like while all these blocks are optimistic while the el catches up with the state of these payloads of corresponding

payloads in parallel and after the justifiation point in the store has bumped uh it appears that uh a portion of blocks that actually do this justification appears to be invalid so cl have to remove these uh invalid blocks from from a from the block 3 and after that we can get into a situation when uh there is no um branch that realizes that that there is no viable branch viable in terms of ffg meaning that there is no tip

in the block tree that has a justified checkpoint equal to what we have in the store so and the question is what to do in this case so what what how the node should behave uh first of all uh yeah like uh there could be a different solutions like for example we can roll back the justified checkpoint uh at the first glance it seems okay but it can may lead to the surrounding so it's it's potentially dangerous and probably the right thing to do here is to keep the node optimistic uh and uh and keep note like in sinking mode

and wait for more blocks to come from like a valid viable chain uh coming from the network and then uh notes uh stick to the um these viable chain syncs up to the head and then get that gets back to normal operation so this pr is just about like if this kind of situation happens and the due to optimistic sync and uh invalidation of optimistic blocks uh a big note uh stuck in this like position where there is no uh viable branches in the block tree it should just keep being optimistic that

that's what it is uh the the alternative uh opinion on that alternative approach uh exposed by kodus is that we should address this situation not only in optimistic case but in all general case uh like it doesn't matter how we get to this um but we should be uh like uh beginners should like behave um correctly around the situation i know photos want to give uh his opinion on that

Potuz: no i i kind of agree with you it's just uh what i would like to see is some some

sort of standard of how we uh we have to act so i think currently perhaps adrian if he's here can correct me but i think uh taco would revert back to the justified checkpoint and accept it as head so that they can continue importing blocks i think paul was about to set up uh to stay with the invalid head and being able to continue accepting blocks this way prison hasn't solved this issue yet we don't really know what is the best way and uh what i kind of see on both texas and lighthouse approaches is that uh if they reboot the node they might be gossiping blocks that they knew they were invalid

prism removes invalid blocks from the database so so that once we know what block is invalid we're not going to go sip it back yeah i have no feelings i just don't know of a good solution to this problem yet

yeah great [Music]

Dankrad Feist: um yeah like uh sorry i missed this castle right now before because i uh my connection dropped but um can you say like you mentioned something about uh having to revert to justified nodes like do we consider uh justified blocks from optimistic sync actually as

justified like before we have actually verified that the chain is correct

Danny Ryan: nodes generally do so to not disrupt the internal machinery too much um this is a critical failure this is the node that was in optimistic sync and but there was either an attack or a chain split um i mean the general the general uh notion since the beginning of this is just that is a failure and likely requires manual intervention um so the the answer the question is what do you do until somebody

manually intervenes i believe

Dankrad Feist: right i mean i i guess my question is is it possible that a node votes

um votes with the source being an invalid

Danny Ryan: no a validator should not be voting on

right something that's optimistic so okay so so so that what is [Music]

Dankrad Feist: why would you slash a database prevention oh like okay as in you will be locked out

but yeah being locked out is different than flashing

Dankrad Feist: yes uh well yeah okay [Music]

well i mean in practice in any situation where you're locked out you're probably going to be part of the inactivity

Danny Ryan: sure well not you can have an accidental sign message that isn't correlated with a lot of other people

Dankrad Feist: yeah but if the block is justified then that's probably not the case

Mikhail Kalinin: i would like not uh rely only on the session protection here i mean the note that got into this state it just must not must not uh test at all

Danny Ryan: yeah um that's that's defined in specification

Mikhail Kalinin: um yes but uh but when you're invalidating optimistic blocks you're like yeah you just have no head no viable hat that could be the case

Adrian Sutton: yeah i think it's just important to clarify that in this case the justified checkpoint block is valid like it's fully validated but we've had a sequence of optimistic blocks that caused us to update justified checkpoint and that sequence turned out to be invalid so we shouldn't have updated a jackfruit checkpoint but we're kind of stuck

um because we we aren't able to revert that in clients currently

Danny Ryan: right

Paul Hauner: so does the justified checkpoint need to be fully valid it can be optimistic right

Danny Ryan: well it needs to be fully valid if you're going to actually act on it but

Adrian Sutton: yeah so so it might be optimistic in in the normal case it would be and then this all kind of resolves itself and we're still optimistic regardless it's the case when the justified checkpoint is valid but we had to fall back to it because of an update to justified from a sequence of blocks that turned out to be invalid that we wind up in this case but we might be attesting based on

changes from optimistic blocks that we've since provided even though our head isn't optimistic

Danny Ryan: so i would argue that it's important that you're not attesting to optimistic things and that this is an acknowledging this is a failure case and then the failure case i don't know needs to be standardized because the failure cases i either stuck and i have to have user intervention i have more sophisticated machinery and i can try to work myself out of these optimistic blocks which by default i think a lot of people don't want to do but is not necessarily a bad

thing to do if i'm if i'm able to actually go and revert this machinery knowing that if i wasn't a testing i'm not actually doing anything bad i'm just trying to get out of that reality um but i i don't know if one or the other needs to be the exact thing that we do

Mikhail Kalinin: the problem currently is if and not currently follows the stack it will have it will have the justified route as the head so it will just revert the head back to its justified route which was like pushed by invalid blocks but it's not a problem it's like a valid justified route with

enough attestations to justify it uh the problem is that uh there could be um some um some honest blocks on top of this route as well um and yeah if if like majority starts to vote on the justified route right um and i don't know builds a block on top of this so some some of the honest block a version of honest block and the real yeah but definitely it's like a very much case scenario i know probably i should like put more um contacts to this uh pr i mean like the example of what can

happen and what the situation that i'm trying like to just right now i just

Potuz: say it's not so much edge case if uh given the fact that suppose an attacker can actually get some nodes into optimistic mode so this is along those frames of attacks that get worse once they manage to get you optimistic because they only need two blocks if they get the last block of an epoch and return a syncing and then the next block of the epoch and turns out to be invalid uh then they can just pull they can just trigger justification of uh of a new of a new block so only only two blocks is enough to trigger this

Paul Hauner: i don't want to get into details of how we can do this here um but it's a little bit more difficult than that with some some changes

Mikhail Kalinin: yeah let's probably take the discussion into discord and uh this

vr

Potuz: so i think prison is going to do what uh what lighthouse uh is thinking on doing on keeping an invalid head uh within four choice and just ignoring the fact that in this situation we're importing blocks that uh where were four choices in an incoherent state

Paul Hauner: yeah i think all the approaches are pretty reasonable um that are on the table as well um techies one i don't see any problems with it i think all of them if all the nodes did it all at once we'd be in a lot of trouble but um yeah i don't see any problems with either way at the moment

yeah

Adrian Sutton: i'm not a huge fan of trying to have an invalid head it's just it's very hard to reason about what the knock-on effects of that will be um but uh i need to take a bit of time and look at whether we can be sure to stay in optimistic mode i think that's quite reasonable in this case

um [Music] it's probably doable

because i think

Paul Hauner: my latest thinking with regards to optimistic is that um the node doesn't care it doesn't distinguish between optimistic and invalid um the only scenario it cares about is trying to avoid the valid ones when it's choosing ahead which you can't always do um but yeah i've been trying to take that approach now where invalid and optimistic are the same thing because you can't trust them and all we do is try and fork around the invalid ones if we can

Potuz: that wouldn't work for us because we we do differentiate between them we actually remove everything that is invalid so uh

we're gonna have a a a root for head that is not no longer in portuguese so we can treat those that situation that's a very special situation where our head is no longer in fortress as optimistic but but it would be something new

Paul Hauner: yeah leaving the invalid ones around and just keeping isn't optimistic is i don't know if it's it's generally easier but maybe it's harder if you haven't gone down that path

Potuz: it's it's easier on broader array but we have another four choice implementation where it's really trivial to remove everything invalid you just remove one node in a tree and automatically removes

the whole subtree but now we're getting technical

Danny Ryan: okay um let's circle back and discord this issue and refine it thanks for the details and discussion

with phil

Danny Ryan: okay any other merge related discussion points for today [Applause] okay

great moving on to me boost there is an open issue around mvv boost during the transition i think there's been some general agreement on the past call or two but alex is there anything else to do before we merge this

Alex Stokes: i don't think so let me grab a link here uh basically on the last call we agreed that we would more or less have an embargo of using mev boost through the merge transition and when we would stop this quote embargo would be once we have finality so this pr reflects that and i just wanted to call it out before

it gets merged in uh i think a number of you have approved it on the pr so i guess this is your final chance otherwise that's what we'll move forward with

Danny Ryan: okay great and uh immediate liveness discussion alex looks like you have a doc and maybe some um quick discussion points you can get give us here and then we can um do some review outside of the call

Alex Stokes: sure let me also grab the link here so let's see

what i'll drop in the chat is a very high-level sketch of a proposal so the proposal wants to address a problem and the problem is the following there's essentially a very extreme failure case where nodes are using immediate boost post merge and what happens is there's this actor called a relay who should release the block after it's been committed to after proposer has signed that this is the block they want to propose and this relay could fail to release the data they need to and suddenly there's a liveness failure on chain because the block just can't be assembled

so uh you know to be clear we don't expect this to happen but in the event that it does we should maybe have some mitigation what this would look like on chain is suddenly a bunch of slots are missed and so this proposal says why don't we just use this as a circuit breaker condition so if there's say like five blocks in a row that are missing there's luckily stop using that boost uh this might be you know this could lead to a lot of false positives because there's a lot of reasons there could not be blocks suddenly but it's sort of like a safety thing where it's like we should at least do this and maybe do some other things as to figure out why there's certainly no box uh but yeah so i wanted to bring this point up there's this document here

the document if you review it it's like again i think pretty simple and straightforward and part of that reasoning is because uh ideally this is something implemented in beacon nodes before the merge even happens so pretty soon um there's a bigger conversation around how sophisticated these first sticks should be around when we you know stop using that boost or when we re-enable it and things like that and yeah maybe uh you're welcome to leave comments on the doc or maybe actually taking them to the discord at this point would be a good place to continue

Danny Ryan: right so it's easy to say you produce a block um and you

commit to produce to some bid and then you don't get the full block it's easy to know locally okay well i shouldn't use that relay anymore but i also might be one validator and might not get another validator for and i get another proposal for quite a while and so that information isn't actually useful to everyone else if it's something systemic that's happening and then there's not really a way to demonstrate that easily to your neighbors maybe there is with some sort of gossip and timing analysis and things like that but uh but we're not there so instead circuit breaker that makes sense

miguel i think you raised your hand

Mikhail Kalinin: yeah um i'm wondering if we can increase the uh maxilla missing slots to something like i don't know 16 probably um and what what like the you know what how to say what we get if we have this value too big i mean what downsides we can get into yes for the current value it's it's like pretty easy to be exploited by uh adversary like i don't know that earns like 10 percent of the stake uh they will just have a chance to to to just withhold these five blocks in

a row uh then publish them after some time when everyone has switched from map boost to local els if there is there is any gain from there is any uh payout from from this game from this behavior

yeah they're probably because they can have mg boost on hard-coded i am not oh i heard someone talking very quietly in the background

Dankrad Feist: um would it really be like i mean how would that work if you release those blocks i mean

it feels like

Danny Ryan: you don't even have

to release the blocks you can just miss them and everyone else turns off mvv boost

up your yeah

Dankrad Feist: but then you but then like you missed five blocks i mean that's that

sounds yeah

Danny Ryan: you might have multiple you might have many epochs in which you're the only one successfully getting

an ev

Dankrad Feist: wait how long are we turning mev boost

[Music] how

Dankrad Feist: about making it an exponentially increasing

um recovery period like first time you do like 16 blocks then 32 blocks and 64 or something

Danny Ryan: yeah i think that's reasonable i think we're also probably we're going to be balancing simplicity um just if we want to ship something like this it would also the other one is to say model your adversary and then find the number you know what what size adversary are we talking about and what number of blocks is it uh diminishingly small chance that they're gonna get in a row

yeah

Mikhail Kalinin: and just i i think that it not necessarily you miss those slots you may just withhold these blocks and

release them later so you will not miss the opportunity to propose and all this stuff not sure how this is able to do but

i mean one down side of this is like even with five blocks this means mav boost can uh can only deliver every fifth block and still everyone will continue using it like that's also something we consider here and that might just be an argument again

Sean Anderson: um so we've actually implemented this in lighthouse or at least something similar um and generally what we were thinking

is to allow whatever the conditions for not using maybe boost r to be configurable about the user um and then like i think generally maybe it'd be better not to strictly specify the exact value so that they aren't gameable or at least like harder to game and then we could even maybe have like uh diverse like defaults across clients about like oh is it five slots in one client and like 10 in another something like that

Terence Tsao: yeah i think i agree with that i think much of it should be client implementation detail just because i some policy might be easier to implement on one client versus the other client

but uh but i do think like every client should like implement something just to prevent it

Adrian Sutton: i think the concern i have here is that we're adding a bunch more complexity to try and avoid a real corner case um and when we've done that in in various ways in the past you know like doppelganger support we've been seeing bugs where it kicked in when it shouldn't have um which then cause kind of more problems so i particularly be keen to keep this simple um as simple as possible um it's not so bad because you are

falling back your local el but i can kind of see a bunch of problems coming out just because we're trying to track a whole bunch of new stuff or have all these conditions

martin yeah

Martin Holst Swende: um i disagree with adrian what you said about adding complexity because what we're talking about here is basically a switch that disables the integration for mmv boost and in the way i see it everything should continue working perfectly if maybe both boost goes offline if all this you know central

things or if all the components that are not layer zero goes down uh the platform should still continue working uh so i don't i don't think if we if we think that maybe boost going down will cause us lots of problems then we're in a bad place so and based on that reasoning um i don't mind adding uh complexity that may cause there that maybe boosts to start fading uh if we you know yeah so i think it's a good thing if we add this circuit breaker to disable it and i agree that we could

do different choices in different clients and we could even do some like randomization in individual nodes so they have different levels

for what the circuit breaker is at that's it

Danny Ryan: i generally agree i mean right now there's probably going to be one relay maybe two and that they can take down the whole network um if there's not a circuit breaker here i do think that the circuit breaker can be simple and i do think if we model their adversary and we pick it even 32 slots it can be simple and it can be you know very unlikely to be hit by by

an attacker

yeah

Adrian Sutton: i think that's one of those implications i'd quite like to see is at the moment there's a suggestion that we've got to count blocks even if they're not on the same chain in working out if we missed the slot or not um which is definitely required for a small number of slots but if we can make that big and um and be able to just look at the state and go hey i've not had a block you know you look at the block routes in the state and go yeah yeah we've missed 30 slots then i'll stop using their boost that would simplify implementations quite a lot because you're not having to track anything new at all

Alex Stokes: is it okay to not have an epoch or

boxer i

Danny Ryan: i mean

and to automatically recover probably once and then we figure out what the hell went wrong and try to make it not happen ever again

Adrian Sutton: mean that's also assuming that everyone sets up meb boost and isn't paying attention to it so um you know like big providers probably see one block proposal go missing and start worrying a lot because they didn't get an unblinded block they have that insight across their nodes um if you're running a lot of validators and then there'll be a variety of home stakers that probably don't use mev boost because it's just too much for them to set up

yeah it's very simple

Dankrad Feist: this is my main concern with having a fixed number especially a large number of blocks that like they i can see lots of conditions where we have quite a bad failure case but people will continue using mvv boost like say mev boost only delivers every third block as a possible failure case and in this case nobody would stop using any maybe boost so

let's stop

Danny Ryan: using it automatically but we would also work on figuring out

Dankrad Feist: right but we would be in a pretty terrible state in that case already like

Danny Ryan: right that's also like an even more particular i mean there's there's failures that can happen and there's a certain amount of like automatic detection that we can do or is worth the complexity of doing

Dankrad Feist: right but maybe maybe the threshold i mean you can also say like less than x blocks over a period of y and that's a much more robust conditions than zero blocks over a period of y

yeah fair

Micah Zoltu: are blocked builders in this version of meb boost um bonded in any way

Danny Ryan: no nor relays

relays are kind of this you assume the relay has the block and can make sure the block gets out um when you're entering into negotiation with them and this is this failure case when it's right in the middle of the negotiation or kind of right at the end where you've committed but they don't then release it

Alex Stokes: the messages are signed so we'll know who did it but there's no like actual like crypto economic bond at risk

Danny Ryan: is there anything you can certainly do reputation around this these messages can be demonstrated and a certain amount of them could mean i don't use that relay anymore but that's also a lot more complex than this

circuit breaker

Micah Zoltu: yeah but i mean i'm kind of agreeing with um what i think the point that martin was getting at earlier is that the someone should not be able to take the chain down for however long you know minutes hours whatever um by getting control of the relay i think that's a that's a pretty bad failure mode like even if you can't automatically recover

avengers

Dankrad Feist: this is the trade-off we're making with mvv boost

like that right but

Danny Ryan: the discussion is can we ensure that the trade-off isn't so bad

in the extreme yeah

Dankrad Feist: yeah sure yeah

adjusting your

Danny Ryan: your assessment of

complexity on like a percentage of blocks you know x out of n rather than a series of missing um is that is that of similar complexity

Adrian Sutton: yeah i mean i think the ideal for me is anything we can get from the current state rather than having to look across walks that saves me a whole lot of complexity from what i can kind of think initially right maybe i'm wrong in that but that's kind of my first take

Micah Zoltu: the actor who has the ability to or the participant in a system that has the ability to withhold the um blinded blocks that's the relay right and in at the merge the expectation is there will be very few relays is that

correct

Danny Ryan: yes there's also a problem with um such a relay you know if you if you always if you didn't know to switch to relay b because no one's demonstrated to you that relay a is bad then you also you know the multiple relay doesn't necessarily help um if that

relays may be lying to you or something

Micah Zoltu: is it visible to all participants the network when a blinded block is committed to and then never revealed

Danny Ryan: i do not believe that those only but they could be

so

Micah Zoltu: so at the moment at least only the proposer knows that the relay was held

there never gave them the block is that correct

Danny Ryan: correct without additional gossip

so

Micah Zoltu: i know we're very close to the merge and so i'm hesitant to ask this but how hard would it be to fix that that feels pretty bad like it should be that if the relay is not uh revealing blinded blocks that they promised to reveal like that should be known to the whole network as soon as possible and so you can see hey it really hasn't given up their blinded block in the last four blocks i'm gonna stop using the relay and fall back until a human intervenes that way we can limit the failure to you know a few blocks in a row until humans get involved

rather than having to continue on for you know 100 blocks or whatever

Alex Stokes: i think it's tricky to make it really tight meaning like only a few blocks versus say like an epoch just because of the reasons we've been discussing but yeah like chris put in the chat uh there's this idea of like a relay monitor that uh i think at least flash boston tends to have before the merch which would handle these like high level you know like the big failure cases that are easy to to figure out

Micah Zoltu: yeah but i feel like we're not really solving the problem if we just say hey if if that centralized player turns malicious because they got hacked or whatever let's just trust this other

centralized player to tell us about it especially when they're both controlled by the same actor in the ecosystem like if you're gonna if someone does manage to compromise the keys of the people the running mvp boosts they're probably gonna get both sets

Alex Stokes: so the alternative then is like to have an ev boost like gossip some sort of fraud proof around this which could be done but it's more of well you're just

Danny Ryan: though because you can't tell if maybe the proposer was just really late in giving it giving giving it back um okay but we're not some of them enough of

them signals that it's probably fraud

Dankrad Feist: well we can't we can there is one solution where you gossip your choice like basically we create a channel for proposers um because of the assigned block header um and uh you can watch that channel and if you see several of these and you don't see a corresponding block then you know that the relay is bad the problem is like creating that i think that's not realistic for the merch i think that's too much like because you'd also have to consider all the dos possibilities and stuff

Paul Hauner: yeah i would generally say that if you're thinking of a solution it involves adding a new topic to the peer-to-peer network then it's too hard to emerge and i'm pretty sure all of them involve that so it doesn't seem feasible at this point

Dankrad Feist: what does seem feasible is like just creating a service that monitors this like like how hard is it to create a like uh a simple monitoring demon that anyone can connect with using their validator um so like uh one of us could run it it's independent from

flashback that doesn't seem that hard like it feels like a few hundred lines

python script can do it

Micah Zoltu: we still need a mechanism for proposers to be able to

get there i don't know if it

Alex Stokes: needs to be automated like it can be manual to like you know facilitate shipping the merch and then an obvious enhancement is to make it automated

well

Micah Zoltu: if it's manual then it means that the potential outage if the relay is compromised is you know hours two days because people are not

Alex Stokes: it wouldn't be days but we also just said we're okay with like an epoch of missing blocks so

Micah Zoltu: yeah i mean i definitely appreciate the the bind we're in we've got the merge coming up very soon and we have a centralized point of failure and we have to decide you know how long are we okay with the network being down if that centralized point of failure fails um until and we have to live with that risk until such time as we have a real solution such as bonding or at least gossiping

intent okay

Alex Stokes: so there's some appetite for a

circuit breaker of some type uh i'll figure out a place to make that more formal and we can keep pushing that separately there's like other different strategies and tactics we've been discussing around hardening memphis so that's good as well

Danny Ryan: okay cool alex are you going to make you're going to revine refine the that dock and then share it is there a place where we're going to discuss this

Alex Stokes: i suppose the block construction channel on the r d discord but i'm open to anywhere

Danny Ryan: yeah that's good okay okay cool let me find my agenda oh your next step oh no live discussion perfect um anything else related to med boost

Chris Hager: um maybe just a quick call for lighthouse and nimbles to maybe wrap up the builder specs implementation so they can be tested in the next golden shutter fork along with the other clients that that would be ideal

uh yeah

Sean Anderson: so

our pr is actually under view right now so getting close

Zahary Karadjov: the same is true for nimbus we are close to merging this vr and we are planning a release shortly after

Danny Ryan: cool thank you okay anything else on mbb boost great any notable client updates are happening xero would like to share okay other research spec or other items

to discuss today and anything at all that people would like to say announce discuss or otherwise

Tim Beiko: uh we have uh oh i'm just gonna say quickly we have a 4844 call tomorrow at 14 utc um if people want to follow up on progress on that the link is in the pm repo

Micah Zoltu: where is the appropriate place to cry about this mvp boost situation offline

Danny Ryan: maybe on the um builder specs repo um you could write an issue about proposed bonding or other things like that and also in the block production channel on the discord

okay

Dankrad Feist: there's going to be no bonding before

bbs

Danny Ryan: meaning any sort of bonding that you would do would without l1 support would need to be it would be opt-in and some sort of extra protocol mechanism that relays decide to do

well

Dankrad Feist: i mean maybe with chain link or something but

it would suck

Danny Ryan: oh yeah yeah you would need you need like like alex said a

committee

Dankrad Feist: like you need an you need an availability oracle for the signed header and um and the main block and that just yeah yeah i think it's very unrealistic it's not it's not gonna happen i'm not i'm not just saying before the merge i'm just i'm saying it doesn't make sense to do this before pbs like mev boost is literally our stop cap solution for that that's how it was conceived and

i am yeah

Danny Ryan: yeah i think we can get better on

reputation and more transparency around messages and a circuit breaker but i also agree that

you know

Dankrad Feist: yeah i think these are great i mean and i think that's totally possible and i think like it's just like this doesn't have to be a like a monitoring service doesn't have to be decentralized we we can we can find solutions for that that work very well like i don't know like i mean if there's something that like at the moment you build a piece of software that any of us can run and that all the proposals

can submit their blocks to and that can like sound around with like relay has missed several blocks then like um why is that not good enough i think that's a pretty that goes pretty

fast

Micah Zoltu: yeah i would i would be significantly happier like that than her situation i'm saying that i do think we should be built focusing on building something and that something is going to involve clients like the client are the ones that need to send to that

no

Dankrad Feist: right right right okay but the client or the client

[Music] has to do should be fairly minimal sure

Micah Zoltu: yeah i'm just saying like i think

this should be a focus like we should be figuring out what that solution looks like even if it is centralized you know make it centralized dish right so anyone can run one of your services and you can point your proposer at n of them and so you can broadcast out to seven different ones run by seven different people around the globe and if any of them report back um you have evidence and they can give

you proofs

Danny Ryan: i think very fortunately that a lot of this actually can be encapsulated inside of the med boost sidecar um and so it's much easier for me boost sidecar to add some sort of like experimental gossip channel to be tracking relays and reputation and that kind of stuff or even be communicating with an external um monitoring service um and it's been

designed that way for a reason right so i you know i can i don't have to go into the internals of uh the beacon node for everything as long as like the beacon node knows to communicate with the sidecar as you know to get extra uh valuable blocks and if the sidecar is not giving anything it can go local um and so i think you know the complexity is very nicely carved out there to iterate and continue to build up and be used okay shall we close the meeting anything else

Parithosh Jayanthi: uh just one more thing the body blog

post is out so please update your notes i think all the client teams have releases and the curly fork should be in the next two weeks

Danny Ryan: very exciting okay thank you everyone very productive conversation talk to

you all soon see you on thank you thank you [Music] [Music]

[Music] [Music] [Music]

[Applause] [Music] [Music] so

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.