All Core Devs - Consensus #122

Transcript

[Music] great

Danny Ryan: okay welcome to ub's consens L

call 12222 um this is issue 900 in the PM repo um hopefully short and sweet but we shall see uh so first and foremost um usually we have you know testing slub nuts but instead we have blobs side car networking updates status because testing and dnets and other things are generally blocked on the consensus layer rework that is currently underway so I'd love to get like a general status update and then we have one issue posted by enrio um around when to serve certain messages that we can talk about um most core devs that's the

name of somebody on the zoom uh which teams are behind there just so we know all teams okay

a lot of teams cool yeah we like 40 people here I think oh hello do you have a video your depth of field is very proba nice

cool

Danny Ryan: um so I would really like to hear from uh the various consens teams on just a status update on the rework the networking rework with respect to the spec changes a couple weeks ago um who wants to get us started I can start it's Sean um so we're pretty much done with development um I think we would need to the end of next week in order toh get through review and finish up uh testing but yeah that's about it

nice

Enrico Del Fante: from the tech team

um work is in progress we implemented the new gossip um validation we are migrating all the structure that are needed to be migrated out we are working on the Builder flow and U yeah working progress on on track with timings

uh for lar uh I

Gajinder Singh: aim to complete this uh in this week hopefully next week lotar should be ready

Terence Tsao: um for prism I think um for the normal path which is with without Builder I think um end of net switch is

fairly doable but with a builder probably need one more week which is probably fine because like builder takes a while to set up anyway

en Rico

Danny Ryan: real quick when you said on track with timings did you generally mean kind of

Enrico Del Fante: yeah I think I guess a couple of weeks um seems like reasonable loar yeah I already apologize nimus anyone from

N I guess we got no one from n yeah I don't think so okay cool

Danny Ryan: um so it sounds like we are not ready to plan Deets but we're ready to kind of pre-plan talking about it again um so at least let's bring it up on acde next week get another status update and that'll inform whether we're kind of uh working on some Proto devet the following week or maybe the week after um obviously devop folks if you want to have keep a closer eye on the pulse

um maybe you can do some some pre-work you know if a team or two want to start experimenting at the end next week I'll leave that up to you that sounds good uh ideally I think it would be very good if we could launch something next week like Dev 12 next week but if we're not going to have more than two clients ready then maybe a week after that and but like end of November is probably like 29th to 30th November is it's probably more reasonable for what's the status on high test do we already have them updated and other clients that are already testing on

Mario Vega: hey yes um so we're basically ready on the M Builder and on the um blobber

side so we ready to start testing the Builder I mean the normal flow the Builder and the blobber um stuff I've added some interesting test cases on the blobber I will try to share them as soon as possible on the R D um basically just equivocating block block heads and all that stuff um should be should be ready to go um so yeah we we are basically ready just adding more test cases that would be ideal over the course of the next week but the basic stuff is is is is ready sounds good so if you can have like a some client branches ready to go by end of next week then we can do hi

test a week after that on Monday Tuesday then we can launch on Wednesday around the 28th then 12 and if that goes smoothly maybe we can for early in mid December sounds right

Danny Ryan: I'm not going to commit to a date but I do think that certainly next week we can start preparing for Dev 12 whether that's all clients or a couple okay um anything else related to

this item Okay cool so the next one's kind of related at least to some of the edge cases that come up in this um and Rico I did just toss a comment on there but can you give the quick on this and we can discuss

yes

Enrico Del Fante: sure um so I was thinking about highlighting a difference that will be introduced by the Nab in terms of usage of block by root and blobs by root RPC requests because before the NAB what happens is that if a if a node misses a block sent by gossip

the moment in which he the node will start using RPC is when first it receives an attestation voting on that block or even if receive if he receives the the next block building on top of it in both cases the most of the network probably has already imported that that block so if the all the clients actually serves only blocks that are fully imported it's most likely that that node will be successful in in retrieving the block by root but with the NAB uh a node will will be able to uh be aware of that

something of an of a block or a blobs that he missed by by a gossip way way before that that this is because we we can receive a bunch of set of messages that all related to the same block route

like

Danny Ryan: there's a native a native race condition rather than like a accidental sometimes race condition

Enrico Del Fante: what do you mean by that

Danny Ryan: like built into the protocols we sending messages that have linked dependencies at the same exact time yeah yeah

Enrico Del Fante: exactly so that that is this is this is perfectly um a consequence of splitting and and DEC cing blobs and blobs blobs

and block which means that potentially the node could decide to uh do a bu root request both on blobs and to blocks a little bit before that but if the client still serves those RPC request only for blocks and robs that are fully imported there is a high chance that those early blocks those early by root requests probably fails um so this is related to a a logic that we already implemented in Te which is a logic that tries to recover when

something is missing and the attestation due is coming and now it's time to to attest so we still give some time to to to to the gossip uh layer to receive the the the missed message but at at some point we try immediately to do to do the B request so I would thinking if um clients could start serving um messages that has been validated via gossip not fully imported yet I think this um the tech Behavior could could receive some benefits on doing that uh

so it will be some early recovery and give more chance to not to to to attest to the right thing um from a security perspective I think like receiving uh things that are validated via gossip or actively requesting for a particular message doesn't think to me doesn't feel to me that this there any particular different

right I

Terence Tsao: I guess I have a question so um do clients wait until they have all the blocks and the blobs assume there's blobs for the block and before

everything then start running the um State transition function because like theoretically you can do everything in parallel like a pipeline right because you can do the consensus first and then the execution can also be in parallel of the consensus and then after that you just wait until you have all all the blobs because that will save you at this like a couple hundred millisecond at Le that's this is what prism does

te

Enrico Del Fante: does that as well so at some point you could have three actually three three three pipelines in parallel one is the straight transition because we we start doing the block import as soon as we receive the block

then there is this pools of of blob side cars that are building up gradually and then at some point we also engage the execution layer so there will be blobs block validation with the transition and and execution layer that does the payload validation but doesn't mean that at some point you really need those blobs to fully uh import the block so if you have the block and uh three of out of four blobs and you're still waiting but you could have some time to to to get the missing blob earlier and

actually import everything before attestation you

Danny Ryan: I mean my intuition here is that like being able to request the message and the responder of the message sending it to you upon the same conditions that they would have forwarded on Gossip is not increasing the attack vector and then potentially removing it from such message responses in the event that it failed full validation um yeah I think it's worth that optimization to keep the net Network kind of lubricated here

Enrico Del Fante: I also think that with blobs number Rising maybe the the the chance that one of those are

missed maybe Rises and uh but this also valid for from from from the Block perspective so we received the all the blobs but we miss the block and we can do an RPC by root um for the block itself we I think we already do have this Behavior where we start serving blobs or the block um prior to import via RPC um but from the uh when we're

requesting we actually are delaying our by Roots request to sort of like optimistically listen on Gossip um because we've noticed that it's more likely that we just end up receiving it on Gossip later than uh we complete an RPC request in time to actually attest um so so generally like I support this sort of I guess uh Clarity and how we should behave in RPC um but yeah I'm not sure how necessary it is in terms of like uh a testing in time but it seems better to try to like seed the network more

quickly I guess um even if we don't have blocks and blobs fully verified together

Danny Ryan: yeah I mean I guess there's two things here one is like how a node should realistically kind of decide when to switch from I'm going to rely on the gossip to actually I should go find these messages and I'm not certain if we should that should land in the spec but the clarity on is it okay to serve messages that have past the gossip validations only to potentially remove them from things you might serve in the event that it fails full validation I think that that is Def my opinion is that's the way to go and if we want to make sure it's clear in the Spectrum definitely cool with that does anybody dissent from that opinion

that like if it passes gossip validation while you're doing full validation it's okay to serve so I will say it is like more complicated to implement um but I do I do think it's better

um Mel is here

Mikhail Kalinin: I just have a question uh which case would try to optimize um so as I as far as I understand a you know a remote Pier sent you a plop and uh you assume that it has also it this pier has also received a block right uh and this block passes gossip validation but then uh this pier must have sent you this block as well so

we are optimizing for the case when this uh push from that Pier uh push of the block has failed and we are requesting it by root rate am I understanding this correctly so the pier tried to send you this block while gossip but did not um but failed to do this only in this case this by root request will make any difference in my opinion

um here

Enrico Del Fante: yeah I was also thinking about doing that oh no no okay yeah yeah that's that's the optimization so if for some

reason you got some kind of networking glitch that um that prevents you to get those messages one of those

you yeah which

Danny Ryan: these are different meth meshes like it's not I I I I didn't understand like the push vers PLL thing here but like if I receive message A and B on the meshes I'm expecting and then they have a dependency which is message C and I didn't get message C it's it's a matter of who's going to respond to my query for message C is it people that have only fully validated message C or is it people that have validated the gossip conditions of message C

do that sense or we on the same page

Terence Tsao: I

think there is a difference right because you can pass gossip but you don't verify the state transition function right so I think there is three levels to it the first level is you pass gossip the second level is you pass State transition function and then the third level is that you have everything right and I think it's probably better to serve only if you pass station function because say if today like attestation is is is bad in the block you will still serve it versus you actually verify it

Danny Ryan: so I mean we we definitely can't gate on the third because uh then it would be very difficult to recover because only nodes that have seen everything would be able to actually respond I'm arguing we

should only gate on the first because arguably those conditions made it safe from an anti-dos perspective to forward the message to your peers on Gossip so it's arguably safe if somebody requests it from you um obviously like you could add the additional full validation but I don't think it really buys you anything with respect to security um and just takes more time

Terence Tsao: it's also easier to implement if it's just the first

I guess from prison perspective yeah yeah the other wayse okay for is is the same

Enrico Del Fante: so if we we have this pool of things that has been validated and pass the gossip

validation so we only have the first and I was thinking about serving only that set of data so for prison the problem is on where they are stored so if we if we go to a in approach where the things that haven't been completely seen and validated are stored differently that those that were already seen and validated then when you're getting a request by root it becomes a little trickier to be looking in two different places for blobs to serve

Terence Tsao: but what I don't understand is that if today you pass gossip you should be able to run state transion fure right after right so there is not that big of a difference and then and then the state

transition function is quite fast anyway anyway

Danny Ryan: yeah but if you're talking about like 200 milliseconds to run it and you're talking about a network that's TR to heal repair itself like in these four second Windows it it that could be meaningful I obviously that's just kind of an intuition and I don't I totally know the impact of of those on those requests but

Terence Tsao: yeah that's a fair point so one one idea that we had uh long time back um was like you guys are like making it kind of binary whether to run the checks or not and uh one idea would

be to not make this binary but say okay every second request we uh we run the state transition function before we uh before we distribute the block or not and if we we can do some like we can use the peer scoring and say okay uh for peers where we have a higher confidence that those are good peers uh we run the we we we forward the block directly oh we forward their block directly without checking it and for peers that are uh lower scored the the probability like you you will the way we implemented it

back in the day was not on go ethereum it was on a different project was that you um I don't know you to normal Pier would have like a 50% chance and on on on every on every block you you receive you you would evaluate it whether um like you would generate a random number between one and two or whatever one and 50 and if it's below 50 then then you would evaluate before you send it out and um the more confident you became in that Pier the higher the the probability is to send the block out without verifying it um so that might and that that would what what this would

give us is basically in the in the happy happy happy scenario if there's no bad blocks on the network no bad attestations we are in this fast mode where we just like we just propagate everything that the um without checking it and if we start seeing uh attestations we will um check stuff and and kind of prevent the the the propagation of of those bad things

yeah I think that that

Enrico Del Fante: will apply only on on on the gossip side for us because I think we we decide to to gossip thing only when we Val pass it the the validation of the gossip but we

don't try to do the state transition and then after that say okay let let's let's simate the message it will be too slow

Danny Ryan: yeah I mean to be fair like on Gossip we do we do have anti-dos checks and those are primarily around the proposer signature and so I I just I think it's okay to remap that to the request response here I do I I do like the designes B you're talking about with respect to to gossip Marius because it's I mean certainly very interesting but I I don't think that we need to hoist that into the gossip here so I mean this could this could show up

in a spec in Spec as should or or or or May respond to queries um you know rather than a must but I do think that it makes sense to clarify this one way or the other enrio would you be willing to open up a PR against the byot uh to at least show a version of this and then we can try to land in this SP next week

Enrico Del Fante: sorry Danny you you're saying to me

yeah I said would you sure yes sure sweet

Danny Ryan: um

that is the last sign on agend does anyone else have anything they want to talk about today um if there's nothing else on the consensus layer um there was a proposal from Peter yesterday uh about uh building stateless clients on the on the execution layer in order to help with client diversity um if we have still some time left it would be nice to just get this thought out there yeah does somebody want to give

the tldr I'm happy to if not I I I can I can give the tldr basically the worst case scenario is if a majority client has some bug and we finalize this bug and everyone who who runs this majority client will get slashed and cannot participate in the Network anymore and um the idea behind this proposal is basically you don't you not only run one client but you run all of the clients um but you run most of the clients in a stateless mode so uh whenever you receive a block you will have one client

that you run uh in a stateful mode and you will execute this client in the stateful mode and uh executing the block on it will generate a witness um and you take this witness you put it somewhere memory map file or whatever and then you call all um all of the other clients in stateless mode with this witness and verify that they also execute um the same block correctly and if they execute the same block the same way or if like a quum n Out of M executed the block correctly H then and only then would you

attest to it and this stateless execution is very easy to implement in in execution layer clients you basically just need to have a different database that you can feed this witness to and your your um your evm will just um take the uh take the state out of this this different state database and um yeah that's basically it and so we um because all of this can be done in memory this this uh this witness database can be can be a memory database

um the witness data itself can be a memory map file um doing this would should be extremely quick um and it will just require some additional CPU time and uh what this would give us is that every client would would execute the block on all different imple execution layer implementations before testing to it uh which would prevent us from finalizing um a block in a a bad block in A minor in a majority C that's basically the proposal and so what we would need to do for that is for every execution layer client to implement a

way to get the witness for a block and a way to execute a block in stateless mode with a with given a witness and um the cool thing about it is uh stateless execution is something that we sorry that we want to Target with u with verle anyway and so if we were to build this right now um as I said it's not it's not hard to build but uh most of this code can also be used for um uh for ver and so we would already have stateless clients these stateless clients they are not really feasible for

sying the chain or these kind of things because the Witnesses in uh M Patricia tries are kind of big or can be big um in these in in some in some cases um but because we're not sending this data over the network we only have it um within our node um uh within our computer uh it's not a big problem so yeah that that would be the proposal um I think we we are going to implement it I think B was also already looking into it and um I think some other clients are also

convinced now hopefully to to also join that that's it an interesting I add here is that I think possible heels it's possible that it's possible that eels is fast is is fast enough um but you could do this sort of verification get deals um so potentially you could actually check have B dat is check as well as the proper execution clients think the block is valid but specification think the block is valid which could be another layer that and it might also work with

etherum JS actually so like even the clients that cannot syain it right now because the state is so big and S loads and S stores are so heavy um we can still use the evm to to verify um verify the state transition of the exec can I can I ask a question so I'm not super familiar with the statelessness stuff but do you prove the validity of the inputs with respect to the pre-state hash because I'm thinking this this stateless client doesn't have any continuity so it we just verify anything like you could give it crap inputs with valid execution and it would say yeah that's fine so how does it

know that it's executing a block that is really part of a change so you you you execute the um you execute the state transition and you mark all of the try notes that you touched or that have been modified yeah um during during the state transition and you prove all of them to the state of the pre-state yeah and that's it and you give to both yeah but it's run on your machine as well like the the client that generates the witness yes or other clients r on the same Mach I mean like in terms of security I I mean like if say G has bug in its database can that break the guarantee somehow like if it

generates a valid proof from from corrupt data um then it wouldn't be able to generate Val to the the Old State but the not one stat Cent doesn't know what the Old State R is though so it could just generate it could generate a proof to old State gr no the stateless client would need to have the have the Old State group so the stateless client does keep track of something between just Stout for the last BL was right it need a chain of State Roots okay but but then we sort of still that's out of the trusted input from

right have TR is not going to pass B you could think that's it's not a real issue I mean for me I'm still trying to understand is it possible to do this without the state witness because it's like you already trust your clients to get the right things inste it over so you need to it's kind of important to do the verification because you want to make sure you're running the code that's going to calculate State rot at the end of the block there was something wrong here but it's not quite the same code because you are reading and writing to different for the data and so it's there there's more overlap than having zero

State proof data but it's not you know 100% like there are bugs that can happen on real clients that would happen yeah I also think that like we could you could consider like not actually doing the like ver statement verification and just like telling with C this was the free dat this is all the accounts and ke storage and what their vales were and here is the post value of every store vote and that would that wouldn't catch a bug if there was a bug in like the verle verle tree um but it would C any bug inm and like I suspect

in practice I feel like almost all of the attack surface and like the scope for bugs is actually inm so if we this in a way that like it doesn't catch like a small category of bug like bugs in the implementation of the vle train bugs like in some of the block building you the captures as much as reasonably possible we could cut or bugs in like stateful cases and things like that hey I just want to interject and give lukaj a second

uh yeah can you hear me yeah yeah okay so uh while uh it's technically all this is correct

I I'm not I'm like not that favorable for this proposal uh because it has few things firstly it increases latency uh generating the proof and then uh checking with other seal clients will increase latency on block uh validation so we are already increasing latency with Cancun uh potentially and this will increase it even more uh secondly I disag I agree it's very easy to implement in terms of um evm and executing evm in the client but all the things related to integrating this with

between the clients and integrating this with cl clients uh will is not that easy and uh I think it's uh the estimations here are too optimistic uh and I would prefer to put this effort to actually finalizing vericle trees and if someone isn't uh isn't like uh looking into vericle trees very carefully uh there was tremendous progress in vericle trees this year uh to the point that I think it's close to completion and vericle trees would would allow us to do it better because uh we will we would have this uh things in

protocol so uh we could have multiple clients like doing this at the same time uh without introducing any latency uh more than just the vericle trees because there might be a bit of latency just by Distributing the the witnesses but it's something we probably want to do regardless so uh putting uh a lot of effort and I think it would be a lot of effort into that just to throw it away in few months after I think it might be uh just the wrong way uh and the last uh last thing if I can if I can finish is uh how would the chain behave if for example we would

have a bug in one of the clients and most of validators would run this client and G for example and they wouldn't didn't agree on on would the chain just stop at all or would it just not finalize what would be the proposed solution here uh I think the proposed solution work would be for the for the chain not to finalize um and people just not not attesting to the blocks so the CH will finalize um a very

Danny Ryan: there's a lot of nuance to to to how to potentially do that I mean like the version of this to find some balance is like doing rather than inm do in ofm um where you know

it's you can if one fails or something you can still continue forward but if you have some over some high threshold of failure then you don't continue forward um and try to find the balance between kind of safety and an available chain but not just not a testing is a potential major security issue of itself yeah and the um the other things that you so um as I said I think Proto said today that they already have it and they implemented in a day so I think it's not that much effort and it should be implementable that's prod yeah okay yeah

but um we I think the GU team is definitely going to implement it and um that we can see how much effort it is and then we can also see uh how much overhead that is like we don't know we don't know how much how much CPU it is how how big the witnesses will be collecting the witness is extremely easy so collecting the witness because you you have to go through the tree anyway um you you it's just whenever you whenever you uh you touch something you um you also record the touch into a map and then at the end like you put out the map so it's very easy to collect to

collect the witness

but no no proving the witness is harder if you go through uh the snapshot to load things it's uh one call to the database if you want to prove it you have to go through the entire tree so you increase latency

Danny Ryan: another thing to consider here is like this might there maybe a second order effect to at least keep in mind I'm going down the path is that this might help with uh resilience with respect to multic client um State transition but it might actually reinforce uh fragility in other portions of the stack if for example you have less of a reason to switch from a majority client because you get the the kind of the evm state state transition

resilience then all of a sudden all of the other things become you know like potential Doss vectors with respect to database or um crashes on P2P things like that like if if there's less of an incentive to diversify on these other things um you might actually with end up with more fragility in other parts of the deack um than if there were a theoretical uh healthy um yeah diversity CLI yeah very hesitant um to push for this because it's like we're gu and this will kind of understand stronger but I think it's the right thing to do for the healthy Health he

health health of the network um and um like what I think the the most important thing is that we're not going that we're not finalizing any invalid State um this is way more important than um like if if gu has like 50 or 60% of the network um so uh this is I think this is the biggest nightmare scenario and with this proposal we have a good chance of making this worst night time nightmare scenario um quite significantly better and um so yeah just as a heads up it it doesn't it doesn't

need any uh it doesn't need B bance from all of all the teams but I think it would be really good for for the whole of ethereum that we that we have this these these mods just on the topic of um latency I think we could also just only do the proof once per Epoch because I mean I mean the check the cross client check because the main thing is you don't want to finalize an invid block and that's just dependent on the target vote of the other station right so if that Target block has already been checked with all the clients you can sort of attest to whatever head block you like if that head block is invalid it's not going to count towards the cas FFG weight and you're not going to finalize um so this could be something

that we just need to run at the beginning of the AP on the APO transition which is like a 32x reduction the number of times it has to happen again I I I don't think that latency will actually be an issue at all because this is just CPU and most of the at least in gu most of the latency comes from like Lo like patching data from the this yeah and um so yeah I think the the the best way to forward is to for some teams to implement it um that have the capacity to do it and then just to just to Benchmark it see the numbers and see if if uh if the like the full approach with proving everything and being as secure as possible if that is not

feasible then we can we can reduce the the security of yeah

Andrew

I think in in I agree with Lush I think that in general well I think in general it's a good idea but I would do it with v because then we can agree on the standard API and we don't uh have a custom API built just for that

uh but you like I said for generating the proofs you need to you throw out the the snapshots you need to Traverse the tree all the time so for each read so that increases latency in

itself so that's basically the even the first processing is increased latency a lot and then depending on how uh the proofs are moved and distributed which will have some latency and uh then uh there will be some uh execution on the other uh verifying clients and for example if we have a a very compute heavy block which can be attacked Vector again it might be a problem right but uh again feel free to Benchmark I'm just stating

my uh concerns and that I think um yeah the effort is underestimated to make it actually happen on the production level because it's very easy to prototype it we have it we had it in beam sync uh we have it on verical trees right now and I agree that just the evm part is simple you just need to substitute uh uh the data source but I disagree that it's uh like easy on the protocol level to to deploy it on the network okay it's not protocol even but you know what I mean I think sorry was something else trying to

talk no no go like if if if we want to if we want to have the situation where we can continue producing blocks but just stop testing the C currently has no way of understanding that from the e we just have an idea whether a is valid or not and if a block is valid then we're producing blocks and other stations for that block um we almost need like a concept in the engine API of like this block is sinking or like unsafe for a testing but go ahead uce something on top of it if you want

it

Danny Ryan: but then I mean the implications there of having like an adastation of sport

choices something to think about yeah um I would motion to the fact that I think we all at least understand the proposal and can think about it and gu and others can do R&D and bring back to award Dev what they learn um rather than spending another 45 minutes talking about it here is that next cool thank you

Mars

Danny Ryan: other items for today where's the

party I'm currently in a hotel bathroom so it's definitely not here um anyway hope you all are all well if you're in person enjoy and if not I see you all very soon uh take care bye guys there is the off extion l call tomorrow uh next week on during us Thanksgiving all right

bye bye bye bye guys bye bye

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.