All Core Devs - Consensus #111

Transcript

foreign [Music]

Alex Stokes: this is consensus layer call 111

I'll drop the agenda in the chat it's pmsu 809 uh Danny can't make it so I'll be moderating and yeah we can go ahead and get started so first up uh oh yeah I think Mrs Colby deneb and then we'll we'll touch on some some more research forward-looking issues um so to get started with the NAB I think essentially we just want to like understand what things are finally going into the stack um the spec release soon so I guess we'll just go to the agenda there's this issue for eip6988 I think McHale's been driving this

let's see

Mikhail Kalinin: yeah okay so um about six six and nine eight eight um so um the problem so this CIP is about preventing that a slashed whatever is elected as a proposer which is basically uh prevents us from having a lot of empty slots in case of mass slashing and the one of the problems that we have encountered during work on the CAE was that um basically this EAP changed the brakes

the invariant of the proposer shuffling for an ebook so it can change throughout the ebook um and that that's apparently a problem for other clients and for probably some other toolin and some other balances so I've made it PR which suggests the the change proposed by that plan it basically stores the proposure shopping in state the footprint is not that big and it seems like pretty damn not that big in terms of flag conditioner complexity to to keep it stored in this state so there is the

update proposer shopping which basically accepts the state and I'm going to use the sharpen for the next Epoch and then we read The Proposal this is from from this vector and that's the change and uh yeah it's called every ebook processing this function is called Reaper processing uh what's not that nice about the introducing of this function is that if we are bootstrapping from um from Genesis for instance we have to call this function before our state becomes usable well and before we are going to process any further slots and propose any blocks

otherwise it will just not you know have the proper shaft in it but I don't think it's a big issue so and the question uh main question here is whether considering this additional complexity whether we want to include this uh change into the net so you know if anyone take a look that took a look at this PR it will be great yeah if we can discuss it now so let's do it now or I would keep it for a couple of days more to take a look in those this PR and make like a final decision on that EIP whether it goes in the number not

Alex Stokes: so we do have a precedent of having like

some initialization function for the Genesis state so yeah I agree with you I don't think that's like a big problem uh it says in the description it only increases the extra state that we have to track by to introducing device so not too wild there um I know there's been some like back and forth on this you know trying to achieve the aim of uh this the slash unit variant so yeah it's good to see this here um personally it feels a little bit late to me to add something that is kind of this big because it is kind of like a change to the mental model of the protocol and in the process I don't know if

anyone else is taking a look

Mikhail Kalinin: yeah I'll take that I'll take that

as a

Alex Stokes: no so um maybe we have the pr open you know and uh for another few days and I don't know if uh anyone feel strongly they should they should voice support or not support and we'll go from there I do I would imagine it's a bit too late to

like decide this on the next uh CL call but if that's the next time we get to discuss it that might be the next time

Mikhail Kalinin: yeah probably we can briefly mention it on the next yeah I'll call if it makes sense I don't know

dapplion: so I think here it will make sense to do a study like we did for removing the self-destruct to understand the implications of breaking the dependent route which may not be that bad but I think that's the main question we have to answer here but definitely it feels

tighter than ever

Mikhail Kalinin: yeah so definitely if we want to explore uh breaking this depend to root thing

yeah I would say that this is too late for denounce also

Alex Stokes: waystar can you explain a bit what you mean by breaking the dependent route

dapplion: so this doesn't show up in the spec but in the beacon apis all the duties included dependent route and the idea is if the dependent root of that specific Epoch is not changing because there is a reorder deeper than a number of slots you are good and the duties should be the same so I know a few clients at least I think loadsden Lighthouse rely on that we can get around that and just pull but it will be good because that's exposed

who else may be using that dependent route because we don't know

Alex Stokes: right so I thought really why we have the EIP for this is to basically maintain this invariant that the shufflings wouldn't change so it seems then that you'd still have that guarantee for the dependent route right

dapplion: well my point is if no one is using dependent route maybe we can just break it

Alex Stokes: oh you just mean like get rid of it entirely yeah well but okay sure but my point then is that it sounds like this change doesn't have any bearing on that or am I not following

dapplion: so the eap6988 is a reaction to the simpler one that just

adds a an extra condition in the compute proposal if we don't mind breaking the embedding again um that's that's a much simpler change

Alex Stokes: I see okay so you're saying we could not do this just make you know the few lines change in the other PR and then loses guarantee okay yep um yeah the when I was talking about changing the mental model I think the like having shufflings be you know because we're talking about that they would change possibly every slot right so that to me seems uh even more chaotic than what we have today

so I think we should not go in that direction

Mikhail Kalinin: I know maybe this invariant also used by some other tool in I don't know um and yeah you can have some implications but uh as for me uh not having proposer shuffling swords in the state is and more clear more clean select clear solution and like breaking the invariance if it possible it is possible

dapplion: another point is if you have to process old blocks you need to retreat the shuffling

so yeah having having to press the block the block after that specific slot that's the cost but I don't know I will look into it

Mikhail Kalinin: you would have a state anyway so it's not gonna be a problem

Alex Stokes: okay so I think we should stay focused on essentially what's going to go into like the you know final TM the net release and so it sounds like there's still a few design questions we have some more sort of research we want to do around this particular feature which suggests to me that we table it for now does that sound good

Mikhail Kalinin: makes sense to me

Alex Stokes: okay next up on the agenda uh there is another issue this one for extending the engine API for Cancun I think also Mikhail uh opened this one and yeah anything Michael we should know I think I went the other day and added stuff for 4788 which we'll get to later in the call and generally it looks good when I look at it

Mikhail Kalinin: yeah it's just basically and uh so this spec consists of The Blob extension spec that everyone is familiar with and also with

the the parent beacon block route which is for the other EIP and the other and and last one change that is in this PR in the proposed Lagoon specification for now it's not probably the last one deprecation of exchange transition configuration I would just like to quickly to briefly uh go through it so I think that this pack first of all I think that this pack is just not enough uh I'm in terms of tooling to deprecate this gracefully so basically it's the spec says that

um first execution layer clients must not surface any error messages to the user if this matter is not called if we remove this requirement I mean like if we remove this error message then we can break the dependency between CL and El um so we remove this on the outside and then consistently clients can remove this method entirely uh so what actually how I see the procedure the procedure of removal of this thing is that El clients remove it right away as soon as possible and then we I mean like remove the error message of the methods methods should still exist because otherwise because since the clients will surface error

because of this method will be um responded so yield clients removed the error message uh we wait for Cancun or yeah we we wait for every El compliance to remove this message and then princess the clients also releases this software without this method being called and then after it Cancun everyone can just remove it at any point in time so we will use Cancun as the point of coordination for software upgrade um yeah this is the the way of how to do it gracefully maybe we can just you know cut it cut the court and you know remove it right away but then we will users will see some ux issues by seeing by potentially

seeing error messages one client stops supporting it yeah the other one will complain um yeah so there are two potential ways to do this I would prefer the first one um which is the graceful one but maybe there are some other opinions on that

Alex Stokes: yeah I mean this graceful approach sounds better and was that what you have in this PR or something else

Mikhail Kalinin: now it's not it's not described in the pr because this spirit is into this pack and so I say that this back is not here so probably we should just try this on the alcohol and say and ask el

clients you know to start removing this error message if they're okay with that how does it sound

Alex Stokes: yeah it sounds good to me I got a plus one in the chat yeah uh let's make a note to bring this up on the call next week but otherwise yeah sounds good I know this is something we're going to do for a while so makes sense to go ahead and do it

Mikhail Kalinin: great

Alex Stokes: okay that was pretty straightforward

uh okay so there's a list of VIPs here uh that we essentially well that are on the you know the slave for going into genev um and let's see uh we'll just take them in turn so the first one is 70 44. this refers to uh essentially changing how we process voluntary exits so that they once made are sort of valid forever and this was essentially ux Improvement because before they expired and that was not so nice for people especially in like custodial staking Services yeah so it looks like it's been merged into the specs already

and I think it's on here just to call it out I don't think we really need to rehash but I guess it's worth calling out the next one is 70 45. so this one wants to change how we process attestations and I know that this was around the confirmation rule that some of the Fortress people I've been working on uh does anyone here want to give an overview of this um the pr itself basically just says that you have I believe the current and also previous Epoch to include an attestation on chain whereas before it was just when uh when epoch's worth of

slots so there's basis like slots for Epoch rolling window and now it's basically extended back out to uh sort of rounding down so to speak to both epochs if that makes sense so I don't know if there are any of the conformational people on the call

Mikhail Kalinin: I can try to give an overview on that on that it's not only about the confirmation rule it's also about some other properties in the protocol that we want to maintain under some edge cases so essentially this PR removes the constraint and gives us the guarantee that other stations that has

been um produced in the previous ebook will be includable until the end of the current debug which is important for the confirmation Rule and for some other things as I already mentioned this PR also um fixes the incentives part because previously there was the same constraint on the number of slots equal to number of slots in the ebook for the Target as far as I remember to to give the rewards for the for that so now uh well uh proposal will be rewarded for other

station if it even comes from the beginning of the previous Epoch while the book has been proposed in the end of the currency book so yeah this part is also fixed um and yeah I would be pretty much in favor of pcip to be of this change to be ending up as I think it is straightforward uh there is also the other the P2P part as well so I'm at the stations will be propagate it um yeah so the propagation also changed the from from the slots equal to the number of slots in debug to the previous ebook in the currently book and all these issues produced in the in

these two um are free to be propagated

Alex Stokes: okay thanks for the overview um yeah I mean I don't really see anything blocking inclusion so has anyone looked at this in terms of implementation um that'd probably be the only reason we wouldn't have had with it if there was some issue so does anyone like to implementing this I'll assume no if no one speaks up

Jacek Sieka: okay uh I have a quick question does

this affect the aggregation subscription in P2P at all like which subject you're supposed to be subscribed to for how long

Mikhail Kalinin: I don't think so

Alex Stokes: yeah I think this is mainly a state transition change

Jacek Sieka: um no it's peer-to-peer change as well it extends the time that somebody is allowed to send that stations on a particular Subnet in peer-to-peer

Alex Stokes: right but I don't think it changes anything about the aggregation structure

and that that's actually a good question and the question is that uh in the current like before this change uh how does the aggregation Works in terms of subscription whether aggregators stay for the longer time to expect some attestations from the past or not because the stack is not it does not seem to say anything about this Edge case I mean like even with a 32 slots as it is today

Jacek Sieka: yeah I think that might be a gap actually I think

I think we're kind of unsubscribe early

Mikhail Kalinin: yeah

Jacek Sieka: and then the wrong aggregators will be listening to these uh attestations and they won't aggregate them anyways they'll be lost anyway so I think that's merits at least thinking about

Mikhail Kalinin: yeah so I mean like it does not break uh that much this change in terms of aggregation and uh I think that there is a probabilistic this probabilistic function I mean like whether there is the right aggregator so there is still a probability that somebody subscribed to the right subnet I mean like the current subnet can aggregate

those or or not maybe maybe I'm wrong

Jacek Sieka: well that's the thing that nowadays we will only be subscribing to one aggregate substitute subnet per anymore yeah to be honest I don't know I'm I'm raising the question because I have no idea but but it feels like the kind of thing that would in practice possibly break this or not break it it would just render it points to be propagating these at the stations unless the the

aggregation pipeline is uh in tune with this change

Mikhail Kalinin: yeah so again it doesn't seem to break anything but yeah it's probably probably the problem that we're discussing now already exists in some for some all the destinations

Alex Stokes: okay well so it sounds like we should do a little digging to look into this a bit further and I suppose we'll discuss this on the

next call um I don't think even if we needed to change the subscription logic that it would you know make this twice as complex or anything so I would lean towards uh considering it included but yeah uh definitely something we should take a look at okay so we'll make a note about that uh the next one up for discussion is 4788 so I opened a PR for this basically there was like a feature for this and

now it's been uh in this PR migrated into genev formally um there is some feedback from some of you so thank you for that on the pr and yeah this one's pretty straightforward changed discussing this for a while um I think it's pretty much ready to go there is some feedback here that I can get to but it won't be super substantial uh in terms of the spec or uh a I suppose any final thoughts otherwise we'll get this ready in version for the uh to never release I guess it is worth calling out that uh

there was also an engine API change for this uh in the pr that we discussed earlier from Mikhail

Mikhail Kalinin: um one thing that I commented out in the pr and that from the at the first glance uh I can look a bit odd is that we are duplicating the parent we can block route in in on the CL side so the other way around would be not introducing this into the uh execution payload on CL side because we have the parent route in the outer structure in the beacon box structure

and the advantages CL passes this data into El and El but El includes this parent beacon block route and to the execution payload into the actual execution block um but yeah there was a comment about it and a bit of a discussion and yeah I think that having this work routes inside of um should be about on CL side might be might make sense really from the um like client like my perspective and probably in some other places it can it could be useful so I think it's fine to keep it there um considering that the complexity the

data complexity isn't really really cool here so I'm just yep emphasizing that because probably some somebody else may take a look at that and have some they have a similar thoughts on that

Alex Stokes: right yeah thanks for bringing that up so you know you could say very strictly that we're like duplicating some data in the block with this extra parent roots and while that's true uh the reason it's there is because we essentially want to have this sort of symmetry almost that like what whatever we have in the execution payload in the block is what is passed to the Yale and the El needs us there uh definitely so

I think it's easier to reason about if we just put it there um that doesn't make you all pointed out we could save 32 bytes but yeah um I don't know if anyone here has a strong opinion either way and perhaps if you do you can take it to the pr unless you want to talk about it now regard it seems okay in the chat thank you Okay so

those are the PRS here on the agenda I think the intention is to essentially get them merged into a Genet release in like the next week or two and uh you know that would sort of be our final dinub spec from the CL side so very exciting to see and yeah everyone here please take a look at everything we've discussed if there's any final comments uh especially beyond what we've discussed on the call so far let them be known so with that being said uh the next item here is discussing the Bob counts perhaps this was mainly just a note to

call this out that uh you know I think expect the EAP says that we essentially have a target of two blobs and a Max set before uh there's been some conversation between different uh you know researchers and devs about bumping that up to say uh Target three blobs Max of six blobs and yeah I'm basically I think the cloud here is just to say that people on this call and others are looking at all the data that we can and thinking about you know potential ramifications um please just keep doing that and uh you know join the conversation to the extent that you're able to do so and uh there's a note here we're trying to make an informed minute decision in

the next two weeks so yeah if you have thoughts or feelings about this you should bring them upstairs um I would imagine this will be a topic on the next uh execution call but uh yeah this is just a reminder that it's happening at least it's a topic to happen you know a discussion point that it may happen so don't uh don't let it evolve your plate so to speak

Dankrad Feist: uh yeah and I also wanted to mention for those who went on the Monday court that uh I did more tests at 768 kilobytes and I'm going to share the dashboards in my

chat for anyone who's interested in looking at these um as a summary we didn't see any um yeah problematic Behavior any instabilities um we we did have some problems with our own nodes sinking as far as I know so um the blocks we created some of them while we hit the average were more like 1.5 megabytes and zero megabytes alternating which is actually a stronger stress test but yeah what's our our endpoints the best was a problem

Alex Stokes: okay great and yeah thanks thank God for

filling the charge on those experiments um it seems like from what we've seen there isn't any immediate issue with the bigger Bob size so that's a strong argument to make them bigger uh okay great there's some data in the chat sgar is asking how into the process can we make this decision yeah I mean I would say the sooner the better like I don't think we want to be like two weeks out from the fork and being like Oh you know let's change this just because it'll grip one does I think a bunch of things

Jacek Sieka: uh I'll mention one thing which is

only weakly related but it kind of came out of looking at at graphs around this and and what's been happening over the past six months is that we've gone from practically no reorgs at all to you know a few per hour and we can kind of we haven't like there's no great answer to why this is happening there's a couple of theories um it looks a little bit like like like it's growing with the number of validators it it definitely became worse after uh the complexity increase in Capella

so

Dankrad Feist: well what about the late block reords wouldn't that be a strong reason why we're seeing this

Jacek Sieka: because

Dankrad Feist: before that if even if you get your block and as late as 11 seconds your probe is still going to be um on the Chain whereas now like we walk those blocks so maybe it's not that surprising that this has gone up as two clients have introduced that

Jacek Sieka: yeah it's possible let's say um but if you look at the graph it's kind of growing so that could be explained by you know more people starting to use

newer versions of the clients but it kind of looks very similar to the validators that growing like I don't want to draw any conclusions here really I'm just highlighting it that like when when looking at like experiments um this is like an interesting thing that has changed over the past few weeks uh sorry months and it definitely became worse with capella like on on the spot where we switch to Capella it's markedly higher on average and why am I mentioning it right now because uh well we're packing more and more stuff into the first four seconds before we're

supposed to send them at the station like my gut feeling is that it might actually be an excellent time to rebalance the timing of sending the attestation and the aggregate and I just wanted to sort of fill out the call whether anybody like strongly opposed to say I'm just going to pull some numbers out of my nose there but but like let's send the block at six seconds and all right sorry let's end the attestation at six seconds and send the aggregate appropriately in the middle between that and the next block instead of at four seconds has anybody

explored this and found strong reasons not to do it or to do it um I'd be kind of curious because that would help certainly help reorgs

Dankrad Feist: so I I'm strong in favor of of this and I also have the same feeling that this is an exactly one of our big problems we put one more the first third of the slot I think the interesting question would be if people have um data on when attestations are arriving in the second uh third and when aggregations are arriving yeah if anyone has that data that'd be

very interesting well

Jacek Sieka: we have histograms uh and tell us when all the stations come

can you share

Jacek Sieka: yeah I can share it in the in the consensus Dev Channel later the general trend is that uh or well so there are two things um in respect there is a rule that allows us to send at the stations uh as soon as we've observed the block uh clients are generally not doing this there's like a large concentration of attestations coming in shortly after the four second mark

and then like it spread out but it's fairly like there's a file is a large number of the at the stations coming in the in the second after after the four second marks so like as a client Dev if we're pushing the timing back I would still look into something like I would strongly suggest that people implement this feature where we send the attestation a little bit earlier to spread out the traffic that would help and I am going to post the exact numbers in the consensus tab Channel but I just got feeling based they're pretty good

Alex Stokes: I'm sorry what do you mean it's pretty

good just the ones that you gave as actual numbers

Jacek Sieka: no as in we're supposed to send the attestation at the four second mark and like by five seconds most of them most are the ones that you know we're going to be sent have been sent and result

Alex Stokes: okay right

Jacek Sieka: but again like this is just me looking at the graph I I'd have to I'm going to pull it out in a slightly different format to give better numbers

Alex Stokes: sure yeah yeah I mean data would be helpful I mean I also agree I think this is something worth strong investigation I don't know I would my concern with say

deploying this in denab is just delaying the fork um this is like a somewhat involved change that being said I do think it's like pretty important um so yeah I mean it sounds like we should probably spend some resources looking into this ASAP I don't know if anyone has any thoughts on the Deb and its relation to this change but it sounds like we probably just want some more data first

Mike Neuder: I just wanted to bring up a

yeah go ahead Mike

Mike Neuder: sure yeah just from

the relay perspective um some of the issues around relay stability were around that four second deadline and in particular getting all the relay checks done in time to hit that four second deadline like if a validator sends a signed header to the relay at T equals two then the relay really only has two seconds to get the blog post in time um so yeah I think with with bigger blocks that timeline is going to get even tighter for relays so success seconds might might help in terms of stability there too so just thought that was worth bringing up

Jacek Sieka: uh for relays how much of that is like trying to

you know post the block as late as possible to make more profit versus starting to work on time

Mike Neuder: well the relay can't start the work until the the validator sense of assigned header right so I guess the validator could play like some timing games there's a paper um from Casper and the rig group about this uh let me get the link for it it's called time is money um I think the takeaway is generally validators aren't playing these timing games yeah I'll post a link in the chat um but the relay still has like some amount of latency that it has to do like for example simulating the block takes like on the order of like two to three hundred milliseconds

um and then like receiving all the bytes might take like another 200 milliseconds so like the the latency starts to add up quickly especially if someone calls get payload later in the block but yeah like the Relay can do stuff like oh will reject any get header requests past three seconds but if they have a if they have a valid header that they sign the relay is kind of like obliged to try and get the block published even if the signed header isn't received until like T equals 3.5 or something like that you know

cool thanks for the paper I think terence's parents

Terence Tsao: yeah I just

want to call out about prison also has the release that's coming either today or early on Monday that we do find like an issue where that if today there's the late block and then the substitute for the subsequent slot after the late block that presume will uh delay uh have some additional vacancy um uh blood production and therefore it may cause issues with the relayer and that's why if you're a present validator you may see more reorder or not real or you do a Blog proposed block will have a higher chance of getting New York but the fix is coming so hopefully that you improve things a little bit from the prism side

Alex Stokes: okay that's exciting yeah so zooming out a bit like I think this is actually really important that we look into uh changing these sub seconds or sorry these these so the slot second timings um so yeah let's keep this thread going we'll see again I won't speak to when exactly it's included but uh I do think it's very important for us to to dig in here yeah photos how does you have your hand up we can't hear you you're muted if you're speaking

can't unmute I'm sorry maybe try to get it working and just speak up when you can figure that out yeah I'm not sure there's some messages in the chat but I'm not sure what oh I can I don't know if I can maybe 10 do you know if you have the ability to meet people

Tim Beiko: let me check I cannot I'm getting that you are the only one I don't

wow I know what that means I mean why couldn't he I mean people have been talking

Alex Stokes: yeah but I think Danny's the host technically he's just not here

Tim Beiko: well Daddy just shot a bad POTUS I'll see if I'll give you trick is it

Alex Stokes: wait wait okay he's back very incredibly now yeah I don't know I can chat with you

I'm not sure who the actual host is so uh yeah okay sorry if you want to uh send the message in the chat we can try to go about it that way but otherwise I will keep moving things along

Potuz: oh wait you I can probably be

on muted

Alex Stokes: yeah we can hear you

Potuz: oh good so I just want to mention this thing on the on the on the sub segment that it's not so driven and it's not so clear that you can actually take out time from the first part of the slot because aggregation becomes a problem and becomes an actual problem uh we are we have a very large valuator set that is

getting larger an aggregating is taking over two seconds on a normal computer especially if you're subscribed to old subnets uh the highest I've been monitoring this because we changed in prison the the way we are aggregating and aggregated at the stations and on a normal nuke like mine it would take up to four seconds to aggregate all an unaggregated attestations if you are subscribed to all subnets so that means that you cannot really realistically be a good aggregator if you are hosting more than 30 32 keys on a nuke um so what happens is that very large uh very large

validators can run on on very faster software the hardware but I think for small homesteakers you are not going to get good aggregation if you reduce the middle part of the slot so that leads us to like shifting everyone and taking only uh seconds from the last part of the slot and the last part of the slot I think is it is safer to take some time out of it but the problem is now with the real feature that uh we place a bet before the end of the slot on whether we are going to reorg or not and I'm afraid that we're going to see a lot of split views if we make this last part very smaller

so I think that it's going to take a long time to actually get these numbers correct and getting good experimentation that actually vouches to how long we can increase the first part

Dankrad Feist: uh so what I don't understand is why is the relevant number all subnets why isn't it

because

Potuz: because if you so the amount of attestations that you need to aggregate is going to depend on how many subnets your subscribed for sure and if you run more than 30 validators you are going to be subscribed anyways to all sub names so by so if you're running on uh on on a on a home computer you can't really realistically run more than the two or three

Dankrad Feist: it wouldn't be you wouldn't be in

aggregated or something so

Potuz: well but you are an aggregator quite often and you get all all this uh I mean you are going to get all of this uh many more attestations and aggregated attestations that you are going to uh to need to aggregate so by subscribing to all subnets you're gonna be getting a lot or more of unaggregated attestations

Dankrad Feist: right but you cannot do all that work in parallel right like presumably if if you're if you are running tens or hundreds of validators you know don't just have once you to run

Potuz: yeah the bottlene is even we are parallelizing it and the Border name is is in is in this in the blst library and

it doesn't matter like I see Terrence asking whether or not clients are aggregating uh all at once or on us they come I Benchmark this and it doesn't matter it doesn't make much difference Lighthouse Aggregates as they come and pressing just that grades them all at prescribed times because the the number of additions that you make is exactly the same it doesn't really change anything so so I I truly don't think that we can't really subtract time from the middle part of the slot we can measure it and we can try to Benchmark it but I would clearly see a degradation if we subtract parts from the middle part of the slot and I think measuring the split views that will come from subtracting part from this larger the last part of the

slot is the pain so I think that we should keep our minds open that if we are forced to increase the first the first four seconds then we may need to increase the slots

Dankrad Feist: I I didn't understand why you can't parallelize it sorry that doesn't make sense

Potuz: no we can paralyze it we are paralyzing and anyways we're getting a lot of we are getting four seconds marks so the the typical aggregation for my computer is about 200 milliseconds but then from time to time when there's a missed block for example on you need to aggregate more then you you get up to two seconds and I'm subscribed

Dankrad Feist: well I do need to aggregate more when there's a message

Potuz: well because you get a

lot of attestations that weren't included before that you if there's a missed slot you you have the aggregate the the attention from the previous slot then you need to aggregate with the applications from this Cloud to include more attestations and my computer takes up to two seconds

Dankrad Feist: that's that I don't understand because the previous ones you could already have done in the previous slot

Potuz: no but you still need to aggregate with the current one so so so you run there's always later the statements as well

Dankrad Feist: sure but these are you can just add right you've already aggregated the one

Potuz: well the algorithm to add is not so simple it's simple to add when you only have one bit but then you need to start aggregating Aggregates and and it's not trivial

Dankrad Feist: no you don't I don't understand

Potuz: yes you do okay so I have like algorithms

I have

Dankrad Feist: I had like 200 aggregated signatures now I received 10 more that were late

Potuz: so yeah but then the problem is that you're receiving 10 more that had like 10 two of them are with intersection with the 100 that you had before and you cannot and you've had like a group of like like seven or ten years

Dankrad Feist: okay why do you have to aggregate the Aggregates I didn't know where to

Potuz: because you want to have a better block

Dankrad Feist: yeah okay

Jacek Sieka: uh foreign compared to numbers at least that sounds a bit more

uh your numbers seem on the high end of things

Potuz: yeah so I'm giving you the worst cases so my computer takes takes very little on the on the on the app so the biggest chunk is in aggregating the one beat once uh because we take them all at uh at eight seconds and we aggregate them all at 10. and uh I'm sorry we take them all at four seconds I have eight and uh and that one takes on my three validators is taking about like 25 milliseconds normally but it gets to two seconds on by

locations

Jacek Sieka: so I'm looking at some numbers here actually

um and specifically I'm looking at the delay from the start of the slot when at the stations and Aggregates arrive and this was basically the number that was asked for before and just eyeballing it like it's in the 97 Range that both at the stations and aggregates are in two seconds after when they're

Potuz: well but this is a different issue so I'm talking about different things so we in order for us to submit the aggregate at eight seconds what we're doing is start aggregating before so that their

aggregate is already ready at eight seconds so so at eight seconds is our deadline to submit the aggregate so if so what we're doing now is before eight seconds and this is suggestible by the user we aggregate them all and you need that before to have enough time so that by eight seconds you can actually send an aggregate because at eight seconds we're gonna send whatever the node has because that's the deadline so all right I mean it's always going to be early

Dankrad Feist: I still I still feel like you're misrepresenting the issue because if like valid notes with two validators can manage in that time

then I feel like that's fine that's great because those are just going to get their aggregations in and we don't need the others we don't need everyone to aggregate we just need someone to aggregate right and so you know people who run more validators actually need larger machines to run them then that's not the end of the world in my opinion

Potuz: well I I do think that this is centralizing Force if we're like going to have like people that are be able to be homesteakers can only take like one or two validators four seconds

Dankrad Feist: you were talking about not not like you were talking about someone who subscribes to all validators all

right yeah

Potuz: that's correct but this is the way that we need to when we submit our clients with default values we are typically looking at the work situation which is a validator that runs at least 30 keys and there's many of this and those are going to be subscribed to all subnets and this is what needs to be our default so our timing is going to be this regardless of whether you're homesick or not

Dankrad Feist: I think it's okay to require someone who has who is running 30 validators which is like uh what's like uh um a few million in capital I think they can afford a machine with a few more CPUs to make this fast enough

something to note here though is that

aggregation is not incentivized and so we rely on aggregation but we don't actually incentivize it the only implicit incentive is that you aggregate your own View

Jacek Sieka: I mean there's another point which is that when you're running 30 validators you're not aggregating 30 uh subnets you're aggregating much viewers there's 16 aggregators for every subnet so your child

Dankrad Feist: yeah we're really talking about the node that's running a thousand planets or something that would actually need to aggregate

Jacek Sieka: yeah my point is that like with 16 like the the number is fixed the number of

aggregators are uh subnet

Potuz: that the blocks are full if you have like many more Aggregates that are worse Aggregates like what a validator with only two subnets would have Then You're Gonna Fill the blocks with less attestations

why would a

Dankrad Feist: why would a node with two supplements have worse aggregates

Potuz: because they see less at the stations note the subscribers they get less peers yeah but smaller nodes see many less peers and many less unaggregated attestations than larger nodes that are getting much more peers

Dankrad Feist: is that true

the

Potuz: my node is a much worse aggregate

aggregator than any that the node in the in the in our in our prison like uh kubernetes

Dankrad Feist: I don't understand why would if I'm subscribed to subnet one why would make subscribing to subnet 2 as well make me see more attestation from

something

Potuz: no it's it's it's um it's also it's also dependent on the number of fears that you have if you're a homesteaker like myself on a bandwidth of a home that I am restricting that my number of peers to be I don't remember it was 30 or 50 now by default then I see many many less attestations that someone running with 200 viewers on a cluster those

Jacek Sieka: that's not how the protocol works

Dankrad Feist: so yeah

Jacek Sieka: the number of peers is completely

irrelevant the only thing that's relevant is the concepts of mesh and that one is kept that let's say 8 12 depending on if you look at average or Max so you can be subscribed to eight periods and see the exact same traffic that if you're subscribed to 200 pairs doesn't really matter your your aggregate uh when you're creating a block is Created from listening to the aggregate Channel not from listening to the attestation Channel typically so the aggregators are doing that work for your basic calendar 16 of them right so

Potuz: no no but I'm talking about the aggregating the one feature stations which is when you are an aggregator not when you're aggregating aggregates

the the largest chunk for us is aggregating one bit attestations

Jacek Sieka: yeah and the risk of you being one of those guys it's pretty small that's that's the 16 aggregators that exist per subnet and you're never subscribed to all the subnets and again this is not a function of how many peers you're connected to that is completely relevant um I think this deserves some more investigation and I think we should take it offline and we can go through uh the flow but but like um it certainly merits investigation like in-depth investigation of all these

issues if we're going to change these timings

Potuz: and I'll post it maybe you're in the channels these benchmarks that we have because we changed these algorithms because of these numbers that we were seeing both in my clusters and clusters and common computers and and we were seeing like very large times for aggregations

as

Jacek Sieka: well yeah yeah I'm just saying that like your peer count is not relevant that's not how the protocol works um but changing the timings um the other thing I wanted to say actually was that we don't have to put all the time from the um you know the point where we do the

attestation or the point that we do the aggregate we can we can play around with that a little bit like it doesn't have to be even ly divided or now it's evenly divided and we want to divide it in a different way but like it doesn't have to be you know uh six

Dankrad Feist: I mean it's all right it would even be proper whatever plasma to make this flexible right to not have a 100 fear division between other stations and aggregation

Jacek Sieka: yeah flexible I don't know because that feels like something that somebody could exploit but like it could be two women six or it could be three and

three and whatever

Alex Stokes: right right

Dankrad Feist: so I mean even the ones even the one second is increased on the first third I think would be huge because ice um and my observations there are already under normal networking conditions slots like 3.5 seconds in uh like it's not rare to see that um so like if you add one more second that's actually that's gonna have a huge benefit to the network already

Alex Stokes: yeah I think we can all agree it would

make sense to have more breeding room uh on the front end and so the question now is just one of those numbers what should they actually be there's a lot of data people have been referencing we should uh I think moving this to the consists or some async channel is a great idea obviously there's like this is important and we should keep looking into it but yeah we need a lot more investigation before we can just say oh let's make this six seconds let's make this two seconds or however it ends up yeah so let's uh keep the conversation going but we'll take it offline from so next up uh we do have an agenda item to discuss this proposal uh Mike do you

want to talk about uh the max effective balance change

Mike Neuder: yeah sure and actually this is like kind of flows really nicely from the previous discussion because the goal of the proposal I'll link it in the chat is to reduce the validator set which hopefully would help with aggregations kind of not only reducing the validator set as currently but also slowing down the rate of growth for for the new incoming validators so I'll just give kind of a high level overview of of The Proposal we have a few docs that I'll I'll link that kind of outline the pros and cons and then maybe we can just open up to discussion and as far as some of the design decisions go so yeah the the kind of tldr of The Proposal is increasing

the max effective balance so this this doesn't change the the 32 each minimum balance to become a validator but it allows validators to go above that we've kind of proposed um 2048 eth as a potential upper bound we don't want to make it Infinity um as far as how big a validator can get but yeah uh going up to 2048 we think could be like a reasonable choice um some of the benefits we outlined from the roadmap perspective we talk about how kind of slowing the growth of the validator set will be important for single slot finality dapline brought up the point that um intentionally being blocked on the

current validator set size though that is like a little more of a under debate thing we also talk about the benefits for the current consensus and P2P layers so that's kind of what we were just discussing as far as aggregation is taking a really long time there's a post from Aditya unnecessary stress on the P2P Network I'll link that here so he wrote that before before I published the or before we published the the mods proposal but yeah just kind of talking about some of the numbers of on the PHP layer of of how many messages are being passed around and and all these you know the kind of unnecessary bloat from all of

the the validators and then we also talk about some of the benefits for the validators from the solo Staker perspective it gives this kind of like Auto compounding benefit which people seem really interested in um the the kind of key takeaway here is that with the current 32 8th Max The Sweep just takes all your rewards and and withdraws them so any solo Staker just like would have to redeploy that Capital somewhere else to earn any any yield on it versus if we increase the max effective balance they immediately start like compounding that eth so they're they're earning rewards on on more than just the 32 each that they kind of initially deployed um we also talk about the the potential

benefit for larger node operators who wouldn't have to run as many validators this is kind of like you know a big part of the consolidation would depend on the larger validators actually doing the consolidation there's some a risk associated with it because the slashing conditions would result in potentially like higher slashing if if they accidentally double a test or they double propose so there's some risk associated with it um but in general you know for large validators like coinbase operates something like 70 000 validators that 32 8th cap kind of artificially inflates that number for them and and they've you know various staking operators have expressed interest in

reducing the number of validators that they run um or consolidating the validators into into higher stake but um fewer of them so yeah I guess the that's kind of the high level overview a few of the big questions I think are the design trade-off of the ux versus the complexity of the spec change so in The Proposal we list um we link to kind of a minimal view spec PR let me get the link for that um and it's it's super tiny it's 58 lines added 21 lines removed so you know this shows kind of the goal here was to show how small the the spec change could be but this has some

I guess some ux inefficiencies in terms of if someone wants to get the auto compounding um you know effect they have to actually withdraw and deploy with the new withdrawal credential with this direct zero two prefix instead of the Xerox zero one prefix um there's also this idea that like yeah I guess staking pools wouldn't be able to migrate or to consolidate without pulling their their validators out and redeploying them with like a 2048 validator so they'd have to like to deploy 12048 validator they'd have to exit 64 32 each validators and then deploy a single 2048 one so you know I think it's worth discussing

uh a bigger potential change to the spec that makes the ux better and more more desirable for people and then I guess the other big question that's come up and and dogcraft has mentioned this a number of times is how do we actually get the consolidation to happen like if we make the change it's only worthwhile if it actually results in a meaningful difference in the validator set size and you know if if the ux is so bad and the there's kind of no incentive for them to do it the the big stakers might not do it you know there might be some social capital that they gain by doing something that like quote unquote looks is healthy for your network and improves the overall

P2P layer but yeah it's not totally clear that that they would take advantage of the consolidation and and make it worthwhile so I guess those are the big questions in my mind um happy to open up the discussion here and you know also take questions in the uh in the Discord Channel later if if that's useful but that's kind of the high level

Alex Stokes: thanks I have a question uh just on your last point there is the auto compounding not enough of an incentive to sort of migrate to this regime

Mike Neuder: well so for for big stickers they can take advantage of compounding by just they have the automatic withdrawal sweep

right and then they can just deploy so for coinbase they're running like 60 000 nodes that gives them enough to deploy like nine new validators every day just through the withdrawal sweep so they don't have to really the the auto compounding doesn't benefit them as much as it benefits benefits the little guys there there could be a case to be made that like the because the withdrawal sweep takes a long time like it's you know 40 days or whatever that capital is like effectively dead while it's while it's waiting to get out and so the auto compounding might help them um but yeah I think other than that it's it's not obvious that it's strictly

financially better for them to to consolidate right oh yeah sorry sorry the activation sure yeah that that's the thanks opponents that's the 40-day thing that I was meaning to reference

Mark Mackey: yeah especially since the for Solace takers or small speakers like one of the bigger benefits was was Auto compounding yeah I'm with I'm with you on like improvements to user experience make make it easier for solar speakers so at least it's currently proposed and Mr brought up but um that

max effective balance being 2048 is so high you uh a small state we would almost never experiment to control it would have to like manually withdraw the whole thing at this point to have it withdrawal so I definitely do think that ux is improved there needs to be improved in order for like small stages to take advantage of this and then another Point I noticed in like deposit processing or apply deposit like basically people can top up their if they opt into being a compounding value let's say they can top up their value with the deposits and there is

like a limit that you can only talk about by 32 piece but that's it seems to be a limit per deposit and so it seems like they could I mean they don't have to really wait the activation to they just wait the 16 to 24 hour deposit

Q

Mike Neuder: yeah so this is actually something we tried to cover in the spec um because you're right if they are able to kind of get get around the activation cues and then all of the kind of churn invariants are broken um the way we got around that is just to say like if you top up past 32 East that's just like you can't top up past 32 East basically

so yeah that that's kind of like a Brute Force way of doing it but yeah this this did come up then and it should be covered

Mark Mackey: it will have a limit per deposit like you could submit a deposit within two weeks wait for that to process and then submit another deposit for 32 years

Mike Neuder: so basically how it works is if we see that the deposit changes the effective balance to be greater than 32 East then we the like effective balance of that validator then we just say okay the effective balance is only 32 each so even if the deposit is processed like that

Potuz: so how do you eliminate this thing you're going to have a validator you need to keep track of how far how much

part of this balance in this validator was uh was added in this way or was started correctly this is like are you sure about this

Mike Neuder: no it's just a mechanism to stop them from from topping up past 32 weeks you know we don't care

Potuz: how do we get drunk of this so what how how is this mechanism so

the logic is if these public is known and effective balance is already a 52 we ignore the value of the posited so it's effectively burned

Mike Neuder: right oh so

you're getting

Potuz: oh so you're you're declaring that deposit to be to

being not to be an invalid deposit

Mike Neuder: yeah

Potuz: that's that's something the contract is already deployed that sounds like something that is going to lead to a lot of people being burnt

Mark Mackey: that's kind of true yeah

Mike Neuder: yeah we talked about this you know I think why would someone top up past 32 East like people can burn Ethan a lot of ways why would they top up past 32 East if we tell them explicitly that that eat is going to be burnt like I agree that it would be something we'd have to call out but I also don't think that that would be a normal pattern

Mark Mackey: at the very least you'd have to like in the tooling that are like in the front ends that deal with popping up a

balance you know tell them that it's going to burn it

Mike Neuder: for sure yeah

Mark Mackey: because a lot of people aren't going to know this a lot of people didn't know that there was a leaf oq for the POS activation change like if people don't know these things about the protocol

Mike Neuder: yeah I mean we could also maybe think about ways that the spec like we tried to make this special super minimal and maybe there's a way that we can make deposits part of the like churn limit and and say okay this is what we have the invariant of like one over two to the it's like 1 over 65 000 per Epoch

can't change or whatever so just so

Dankrad Feist: I mean this sounds extremely drastic like I mean I think there's this no go and it's too many this has actually happened a lot of time that people accidentally above 30 degrees so like I mean that that I think is there's no way we're gonna do this it was so drastic for a small mistake um but I also don't understand why why why can you know why would you want to stop people from topping up

Mark Mackey: just getting the activation queue

Potuz: yeah the problem is if you if you get a lot of deposits like this then you get a big change in the validator set quickly but I think we can just churn this and that's it

Mark Mackey: that or like especially something that

would help

Dankrad Feist: right then yeah then deposits just needs like the replace the activation you just needs to be replaced by the positive

Potuz: right and I think we can do this at the same time that we get rid of all of this if one voting because anyways we're thinking on churning those so we can just mix these two things at the same time

Mark Mackey: terms of user experience I mean I know this is historically at the president and my dad a lot of complexity but another a complete other route is is it possible to add a beacon transaction to combine

validators

Alex Stokes: so I think the proposal right now is trying to keep things as simple as possible and adding new types of operations would definitely be more complexity right

Mike Neuder: yeah I think that's kind of the trade-off we keep circling around here is like spec change complexity versus ux you know I think it's worth kind of exploring many different Avenues and seeing what what makes the most sense

Potuz: and I I think if I everyone more or less agreed about like getting rid of one voting and there's already a spec towards that so as long as we we move

towards putting the churn on the deposit queue instead of the activation queue I think we can just mix these two PRS into one

Mark Mackey: that's a good idea

Mike Neuder: yeah that's helpful I haven't been kind of keeping up with the each one voting thing so I'll have to to do a little research there but that sounds promising to me

dapplion: I mean the latest spec doesn't have a cue but it can be brought back

Potuz: um that was just a matter of terminology right because you still have the churn so the issue of the queue being on the states are not in the states I think is minor here

Mark Mackey: and uh

the other is it also like pretty heavily planned or leaned into that we're going to um that we're going to enable execution layer which initiated withdrawals

Mike Neuder: yes yesterday I think it seems like yes that will happen I'm still not totally clear on the relationship between that and this though

Mark Mackey: oh it's just that if we also combine that with this it would drastically improved ux

Mike Neuder: how do the execution layer withdrawals impact the churn but maybe that's a

question for offline but it's not that sure

Mark Mackey: it's just it's just the fact that if you're a small sticker you basically you basically never withdraw and if you have one validator you have to generate 2048 East off of that one even if it's compounded it will take so you need another way of initiating withdrawal

Mike Neuder: you're right but I was just asking if you do an execution layer triggered withdrawal like do they have to go through that the withdrawal queue too that seems okay it would have to yeah

dapplion: I think the point is that yeah

Alex Stokes: the proposal is for exits and it would just move into the execute like it does today

dapplion: okay the the point here is if we can do parcel withdrawals triggered from execution then we could get rid of parser withdrawals automatic personality throws and

Alex Stokes: you're saying you would do that to add this back into the proposal now that Max is talking about

Mark Mackey: well I would I was actually talking about partial withdrawals

dapplion: yeah so the point is like today if you are a solo sticker you have one moderator you need to at least capture some value to pay for expenses and what so not um if if we disable parts of withdrawal signs that you have automatic compounding with the feature that Mike

is presenting there has to be some way for you to extract uh a value fractional value of your radiator without having to exit the full thing

Mike Neuder: so execution layer initiated partial withdrawals not full exits

Mark Mackey: basically right although we would presumably need both

Mike Neuder: right

Dankrad Feist: I mean that would be cool then we could get rid of needing an extra address right we could just switch all validators to this functionality and everyone just withdraws whenever they want so that seems a lot cleaner than than the current proposal

Potuz: one yeah one

problem I see with this is that you you need to bound the the amount that you can actually withdraw on a partial withdrawal otherwise you're gonna get a large change in effective balance in one slot which might be a problem

Mike Neuder: but wouldn't the partial withdrawal have to go so the The Proposal is written to to rate limit the activation and the withdrawal queue based on state rather than number of validators so if the execution layer partial withdrawal goes through the normal withdrawal queue then that rate limiting should be fine right like we just have to make sure that the

that the rate limiting is is correct that's the well yeah no yeah never mind

Mark Mackey: yeah but it ends up like as currently written the the queue is there is no limit for partial withdrawals only full withdrawals because that actually affects the value or set but when you actually have right

Potuz: so the proposal is only to trigger exits not to trigger

Mark Mackey: right but if we do enable partial withdrawals this way like you don't it it now you can withdraw even more than 32 each and still not technically exit your validator so you you would draw on a validator's word without

going to the execute so yeah

Mike Neuder: yeah sounds like some more details to work out but this is potentially promising I guess the one thing about this is it would change the default Behavior so like yeah part part of our design goal for the first spec was like if people don't want to change anything then we wanted to leave that there so that's why we left the Xerox zero one credential alone but if we change everyone like to compounding with these execution layer triggered partial withdrawals then a lot of like workflows would have to be updated I don't know if that's like a

big enough reason to not do it but it's a consideration

Alex Stokes: okay well thanks for bringing this up Mike and hopefully that was helpful feedback thanks everyone for the conversation is there anywhere Mike you'd want to drive further feedback like I guess just to the research post

Mike Neuder: yeah each research post or I think Danny suggested that Discord the POS consensus Channel could be um a good place so yeah I think I should

be pretty easy to get in touch with but would be happy to hear more feedback so thanks everyone

Alex Stokes: okay great uh are there any closing or final comments for this call otherwise uh we'll go ahead and wrap up

Parithosh Jayanthi: um I wanted to bring up an update on the test net call we had just before this

Alex Stokes: yeah please

Parithosh Jayanthi: um yeah so we had the Husky first hoskey test net call uh about an hour ago and we're gonna have the next one again on June 29th with a couple of asks I can link the

um the summary over here uh but one of the big questions that was still open is the current ideas to start with about a million and a half validators so that we have significantly more than mainnet and we don't have to rush to immediately make deposits keep ahead um we're just not sure if all clients think that they'd be ready for such a big value data set a Genesis or such a big Genesis State um so just looking to hear some thoughts on that

dapplion: will we ever reach that in a minute what

percentage of total each Supply stake will that represent

Parithosh Jayanthi: I mean we're at six hundred thousand now with the queue of about a hundred thousand so we're already at seven hundred thousand in a couple of months so we double what we have right now we're also open to starting with a smaller number like a million but that won't give us as big a difference

Alex Stokes: so has anyone on this call like tried I just stayed that big I would say if we can get away with the million and a half may as well

dapplion: but I mean I don't have is 40 million each staked so 30 of total Supply and we do that in a minute that will be pretty crazy

Alex Stokes: I don't know I think if you talk to some of these liquid-staking people like uh they want all these sticks so so

dapplion: all I don't think the numbers are completely taken to be contained and no one can do anything

Parithosh Jayanthi: I think when we started prata if someone had said 15 of all it was going to be State we would also thought it was crazy but we're already here

dapplion: yeah I'm not opposed like seeing the difficulties I would rather start with a big one so we can optimize the clients and get done with it

okay yeah I also

support the nine and a half size

I I have a

Potuz: I have a suggestion as well too I'm not sure how hard is it to do but one of the things that we're seeing on mainnet is that we have uh we now have some validators that are exited and even if we start with a large number of deposits we may increase even the evaluator slides by just adding validators that are already exited on Genesis so that we start with a we don't need to have like a large number of validators sending attestations but the slice itself is still large

dapplion: that's a great point

Parithosh Jayanthi: yeah we can also take that into account thanks yeah besides that we're looking for

um Clan teams involvement to run at least majority of the validators um there's a there's a whole ski planning dock that's been shared on the chat already that uh States what the requirements would be um like what you can expect to get away in terms of machines it is a bit of a investment in terms of money so if so we're looking for commitments like solid commitments from client teams by the 29th and if not possible then we're gonna look at node operators to help help us get to the one and a half million or one million or whatever number we decide on um so please talk to your infrastructure teams and so on and try and get back to

us before the next curve

Alex Stokes: I'm sorry you said that was the 29th of June

yeah

Parithosh Jayanthi: yeah I think that's it on the whole ski topic thanks

Alex Stokes: yeah thanks for bringing it up it's very exciting to see progress there okay anything else otherwise we'll wrap up a few minutes early

okay I'll call it thanks everyone thanks Alex bye thanks everyone

[Applause]

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.