All Core Devs - Consensus #91

Transcript

foreign [Music] [Music] so [Music]

this [Music] [Music] [Music]

Danny Ryan: the stream is

transferred over if you are in the youtube chat please let me know that you can see and hear us this is issue 566 consensus layer call 91 first order business we're going to do these every twice a week for a while to catch up with the awkward dose call and then supersede them in numbering sorry it's a bad joke um today we'll look at submerged stuff some maybe stuff which is related to the merge and then general uh updates and discussion points from there if there are any on the merge

there are a few things then we can leave room for anything else that you want to talk about first uh perry can you let us know what happened to or happen with and how it is going of msf 9. hey everyone um so we had midnight shadow for nine much earlier today

Parithosh Jayanthi: um i'm not actually sure why the estimation was so off this time but the ttd estimation we did last week was hit pretty much like a day and a half earlier and the estimation i ran yesterday was still hit like eight hours or eight or nine hours earlier than expected so it covers um a little bit by

surprise but either ways um we are finalizing on the network but just barely and i'll go through the list of reasons why um so the first issue we noticed was um there was there was a problem with the lighthouse modes where they would ever so slightly fall out of sync and they wouldn't start syncing again until the next epoch where they catch up um the lighthouse team already had an open pr and i think it was almost ready to merge so paul was able to give me like a fixed image for that really quickly and that was patched uh the bazoo nodes i think four out of five of them hit an invalid uh block i

think it's still related to their world state issue um nethermind uh is offline on purpose though uh they ran into this issue on robston and i think uh i think marek will talk about it a bit later but they ran into this issue on drops and they thought they had fixed it but they ran into the issue again on midnight shadow frog eight but after like 12 days of running without a problem um so i think they wanted to track it down so they purposefully ran like 25 of the networks on other mine on like really weird configurations so each of the node was configured differently within different cache rules

so that there they triggered the issue on at least some of them and i think they did so hopefully that helps them figure out the problem um aircon had not synced up to head or some of the aircon nodes had not synced up ahead yet i'm not particularly sure why it's taking so long to sync so from next time i'm just going to be using a snapshot that we have created so we should be able to make sure that they're synced before the um for kits but they are like relatively close so i can give you an update on that later and the nimbus nodes was an issue on my config um i've been merging like a lot of changes we had temporarily for

different shadow forks and on one of them i had a check for if it's another main node it wouldn't run the json rpc snooper because neither sorry nimbus no we wouldn't run the json rpc snooper because nimbus was using web sockets and we didn't need it but in the meantime i had shifted to http but the check was still there so basically all the nimbus nodes were sending the request into the void um so spinning up the rpc snooper fix that and i think most if not all the notes are fine again um yeah i think that's a list of all the issues that we had uh all in all i think like 30 of the network is offline but i guess if

the netherland nodes come back online then a much higher percentage would be back on that

Danny Ryan: got it is there anything in particular that we learned from this either in how to do things or um you know new issues surfacing

Parithosh Jayanthi: i think on my side it helps validate all the conflicts changes that have been merging um and that i'd be using a different method to sync ericon in the future those are the two from my side but i'd let the client demons talk about the rest

okay

a

Danny Ryan: mark i i we can barely hear you can you get close to the mic or try different mic uh right now better

Marek Moraczyński: yeah so uh we had a weird rare issue with three exception that i noticed it before on robsten i think it was the same issue um and uh we fixed a few things uh and we never noticed this issue again for a long time and yesterday after i think 12 days we noticed it on uh maintenance sort of work eight [Music]

and we start preparing uh very different con configuration to get to know what's going on and maybe to trigger it fast a bit faster and it seems like uh reducing pruning cash triggering the issue faster and maybe we have potential fix also but we are exploring it a bit more so

Danny Ryan: got it is this related to merge code paths or is it another issue

Marek Moraczyński: i think it is uh it is uh related to pruning code but it could be on

it could be trigger uh also on main not mine not merge uh pretty much but it is very very hard to trigger this issue uh pretty much and it is easier uh for shadow forks especially and that that is the problem right so i think it is very hard to notice this issue

pretty much i see okay

Łukasz Rozmej: so uh it's a bit related to how because we process a new payload if we can uh a block to validate it and it's related to uh that we had quite random

uh distribution of new payloads uh like going back x blocks like having payload that is uh older in terms of block number uh quite a lot or having multiple payloads at the same level on the shadow forks so uh yeah it's it's a bit more random here and more like frantic so that helps trigger it easier

i see

Danny Ryan: have you all been looking at the antithesis results because they also their platform i think often has get some of these like weird weirder scenarios coming forward did y'all see anything there

no okay um any other client team specific reflections on

maintenance shadow fortnite so one more thing

Marek Moraczyński: i shared probably the reason and we see that uh we are receiving payloads uh in order

and we uh we received one payload out of the order and that probably triggered the issue in nether mind notes but i think generally on healthy network we shouldn't have uh this i mean it is possible of course but uh when we are running all the nodes uh it shouldn't happen so maybe it is also connected to bezel issue that trigger this new payload of out of the order

Danny Ryan: yeah i think dropping that in a gist so that we can see if it's related to one

specific client type would be good and is that are you seeing that on all of your different cl client pairs or is that from a particular yes

okay yes yes because it seems network related

Danny Ryan: rather than um okay yeah if you can put that in a gist with some notes um let's dig into it

Parithosh Jayanthi: another thing to mention is we haven't really had network quite non-finality tests on mainnet shadowful yet but for a period of time i think 60 epochs the network wasn't finalizing until like some conflict changes were made for example fixing the nimbus nodes and once that was done all the nimbus nodes

recovered as expected and the network finalized and it seems like we didn't have any hiccups there so that's good

got it

Danny Ryan: good to know i'm looking at the screen you want to ask lots of questions but i don't think we should live debug this um okay any other um shadow fork discussion points and if not what's the next plan on shadow fork

Parithosh Jayanthi: i was thinking considering we'd want to curly um as the next test net we should have a girly shadow flock next and that should also make it a lot easier to have a bigger network if you want or spin it up more on more short notice we wouldn't have to wait like a week for the early shadow for example because of sync times yeah exactly

Danny Ryan: got it okay and i i suppose we should uh double down on our ttd estimation that we used here uh to understand was it this was the tt was selected like a week ago so we were off by like

half a day and half a day seven days not half a day in the context of a day

Parithosh Jayanthi: i actually know um so the one we selected a week ago was off by a day and a half and the one we selected yesterday was off by about half a day in the um earlier direction on both exactly okay so at least looking at etherscan it does seem like hash rate has gone up but not by such an insane amount

at least okay let's uh investigate that as well

Danny Ryan: so perry on

gourley shadowfork that would be this coming week and that would be midweek yeah i'd say wednesday just to keep it easy for everyone can figs out monday nodes up around the same time and wednesday target something like that

exactly

Danny Ryan: great well if you have um particular builds or config changes or anything like that please please please let perry know

Parithosh Jayanthi: and i also wanted to ask people if they still want me nut shadow for 8 for debugging or if that can be torn down we wait uh a bit i don't know uh one or

two day because we updated

uh notes

Marek Moraczyński: just to see something uh with this three exception of course it is a problem we can switch to uh number nine no couple this is fine

Parithosh Jayanthi: i think it's more useful having it around for you to

debug

Danny Ryan: okay any other questions or comments on prior or future shadow forks okay perry uh you have a discussion of

naming you'd like to bring up as we approach the gorilla merge

Parithosh Jayanthi: yes i think people are eventually going to start asking about documentation etc for what we how we want to treat the future merged girly prata network i don't know early on there was some consensus i'm just calling it curly but that still implies there needs to be like a breaking change or if every client wants to support girly and prata flags and that's what we use in documentation and then eventually there's a breaking change where prata is removed i'm not sure of the approach or even if we want to go down that route but i

wanted to bring it up for discussion

Danny Ryan: and this is because on the consensus layer we have particular flags like if you're doing a network configuration dash network

Parithosh Jayanthi: exactly exactly and it's just easier to communicate to people if if we have like a decision made now so all the documentation written follows those decisions um versus having to change it

right away

Tim Beiko: would it make sense to um deprecate the crater flag leave it like leave it working in clients but mark it um you know mark it as deprecated and um

and introduce the gordy flag and then you know you you k you keep um yeah you keep both as like an alias for i don't know like the next six months or something and then eventually you just deprecate the pareto flag or remove the prayer flag but you have like a deprecation announcement when the gordy merge happens basically

why

Hsiao-Wei Wang: even deprecate it i mean we could just leave it there right

Tim Beiko: oh yeah i mean if you want to leave it there

forever sure and

Adrian Sutton: i think i think the first key question is are clients all happy to have an alias which particularly i think is really trivial and it's a no-brainer

um if that's the case and it will be in the release that we're saying you need to upgrade to for compatibility anyway then the instructions used early and clients decide for how long they want to support prada um i think it's really only an issue if there's a client that that doesn't want to add the gauley alias for some reason um then obviously the instructions would have to be tailored to that client to keep syncroda there's no real reason we need to coordinate the removal of the writer that's just part of the client ui ux

yeah for the lighthouse

Paul Hauner: i think it

should be pretty easy for a saturn alias i'd probably also tend towards leaving prior there as well just for people that already have devops setups like our devops setup where he calls up prada everywhere and figuring out how to change just the network flag name would just be kind of painful

and not much gain yeah same with prison let's start an agreement too

Danny Ryan: anyone from nimbus one chairman i'm here

but i didn't hear what was being discussed

Danny Ryan: that's all right the um it might be easier to communicate to

users to use a girly alias in consensus layer configs so maybe dash dash network corely and just keep pratter there as well um right that's so easy to

do okay great so documentation will focus on just using the dolly flag and the expectations that um merge ready releases would have that flag available thank you

Adrian Sutton: um the only thing i'd probably add is in the docs we should note that you don't need to resync if you use prior like we don't want to cause users a lot of confusion to i think they're actually

switching beacon nodes so if they kind of if there's a note that golly is just the new name for prada and it's an alias that would be very useful

yep makes sense we'll make sure to enter there

Danny Ryan: great anything else on this one

nope that's uh that's all topic

Danny Ryan: okay um tim will you give us the tl dr on your document on the one verse two phase merge deployment i believe most people are pretty familiar with it um so maybe just get the high level

Tim Beiko: yes um yeah so high level is um what we want to avoid for the merge is the ttd being hit before uh bellatrix is

live on the beacon chain um because it causes a whole bunch of issues um and there's three ways we can achieve this um with different pros and cons the first is we have a single release out for the merge which combines both bellatrix and the ttd uh like the actual ttd and because on my net we can fairly well uh estimate uh difficulty um you know we we can plan it such that like it's it's it's quite unlikely that the ttd would be hit before belatrix um and the the pushback against that on the awkwardness call was that well you know a lot of uh the the security assumptions we make usually

revolve around like what if a state level actor wanted to mess with the process and with like the declining hash rate on mainnet there might be a small chance that like this would be possible um if you had a massive massive amount of hash rates um so you know you would get like a simpler ux up front users but like introducing potentially some uh like security uh i wouldn't say like a security issue but like the potential risk of an issue um in very kind of very uh i guess extreme edge cases so the other approach you can take is instead of doing that you do like we've been doing for uh

for some of the test nets where you have a first release go out with like a huge ttd and then um and then once the latrix is live you have a second release which has the actual ttd and then you know for a fact that um that's the second release uh the ttd for the second release would never hit before balatrix because the electric's already live and then the third option you can do is what uh we've also done like on robson is you just do a ttd override um so you know you you same thing you have the first release with a high ttv once bellatrix is hit you just have an override and the the the kind of trade-off there is like you might be able to coordinate and override quicker than you coordinate the new release for downloading um because you

don't you know you know what the flags are in advance and you can communicate them to users in advance and you just tell them plug this value in the flag once you've selected the ttd rather than waiting on like 10 client releases um the downside is you're like asking users to do some more like command line interactions um that they may not have to do if they're just downloading the release um so those are the three options i think if we were to take like the dual release um kind of approach i guess yeah regardless of which approach we take it would be good to take the same approach across mainnet and gordy such that users know kind of what they're

getting uh what what the flow will be like on mainnet um but if um yeah if if if we could decide this today i think it would be really good because yeah then it means we can start like coordinating things for gordy um yeah so i'll

pause um yeah i just wanted to comment it doesn't feel like we need on our main path we need to consider the rest of someone literally increasing hash right by 50 like that someone has that ability and

the best thing that they can do is accelerating the merge [Music]

Danny Ryan: we can't hear you very well but we i i think we get the point um i think this is this has been brought up a number of times and i i i also agree and i'm going to meet you because you have a lot of noise in the background um yeah it's not only if if such an adversary exists such an adversary could potentially do other worse things and if such that very adversary exists and wants to do this they can't do it

overnight they have to do it in a very uh obvious way once these numbers are released and so there's also on the order of a week or more to react but so i think that's definitely the the strong argument in saying that this doesn't really change our the adversary that we're considering here um it maybe gives them one other thing that they could do in this discrete event but that it's very unlikely they could even do it without us noticing and fixing um andrew

Andrew Ashikhmin: um yeah i i think we should go for a single release because the merge is

complicated enough already and like we as developers we can deal with it but to a lot of users it will be very very confusing just to to realize that you need an el client seo cl client you need to uh ensure communication between the two and so on and if on top of that we're complicated by a need for another release and our flag override or it's like it's it's an unnecessary complication that will cause a lot of headache the the simpler the better it's it's like the emoji is already super complicated

Danny Ryan: got it

thanks andrew

adrian

Paul Hauner: sorry yeah yeah thumbs up for me i prefer a one person a one a single release um i think a downside of the dual releases is that it's going to take us more time because we need to give more notice between each release which either means that the merge happens later or it means that we force things to happen earlier that could have happened later whilst the merge happens at the same time if that makes sense

Adrian Sutton: yeah i think the the key in this is that we want to be setting a ttd which means hitting it before the before

bellatrix is activated requires a 51 attack if you've got a double the main hash rate then um that's that's not feasible enough security model by anyone um if you can do that you're hosed you know if we're if we're able to do that i don't think there's any real question that like we're not introducing new security risks and we shouldn't be concerned about that if we're reducing that threshold somewhat then we can talk about it but i'd like to like to see some clarification that we would be reducing that from a 51 attack to something lesser so a lesser percentage before we

worried

Tim Beiko: yeah i think so you're gone oh no i i'm gonna say something okay as long as the gap so basically you know if you think of the gap between choosing these values hitting valid tricks and hitting ttv as long as it's like an equidistant like gap between choosing and announcing the values then hitting bellatrix and then hitting ttd then you would need to double the hash rate if you started mining like on the the second that the announcement is out um to make ttd happen before and then as you get closer to bellatrix that proportion changes right if you're like a week away from bellatrix and like two weeks away or three weeks away from ttd um we would need the like more than

double the hash rate um so i think in practice what if if we wanted to go with this like 50 uh hash rate assumption what it looks like is like you choose you you like announce the releases um two weeks later you have bellatrix and two weeks after that you target ttd and that means that like if you double the hash if you double the hash rate from the time of the announcement you know you might hit ttd before but and then every day that passes where that isn't the case um you need an even greater share of

pressure

Danny Ryan: yeah so it's it's primarily in the if you shift the ratio like if you went 2-1

um on that split then you're you know you need to add plus 50 um and so that's you're changing your assumption a bit um but also not necessarily in a incredibly detrimental way also because it's extremely observable nonetheless also there's always two types of 51 percent attacks one is the existing hash rate uh colludes and 51 attack and others a new amount of hash rate shows up um new amount of hash rate showing up is is

a much more difficult thing from a world resource perspective but not theoretically impossible okay did any other engineering team want to chime in does anybody want to take the counter to this

Micah Zoltu: i still favor the most robust solution but i don't have anything new that's not

already in tim's document i think he did a good job of outlining all the options and the pros and cons

Marius van der Wijden: um i think we should have just like made bellatrix a bit differently so that it doesn't need the ttd to be specified at the fog point and it it doesn't already set it it doesn't oh it doesn't okay

good

Danny Ryan: so it's not like it it like gets baked into the chain or anything like that it's it's a configuration parameter either being said or not i guess you could have done bellatrix in a way that it doesn't need to be talking to an execution layer

um yes to be pinging and then you upgrade it but that's still kind of the same thing that's still kind of like unknown or infinite ttd but then it's primarily error log thing that maybe changes which i think is some of the argument here

Marius van der Wijden: yeah it just kind of feels weird to me that we have two folks that are somehow connected to each other now we're doing like we specify two forks at one point right

but it's fine not going to argue

Danny Ryan: okay um i do believe based off of here and conversations with engineering teams that there is a strong desire to do one-phase approach um based off of the assumptions around the attacker and the engineering simplicities so i would suggest that that's what we do with gourley anything else on this point

this isn't for

gordy

Tim Beiko: but i think for mainnet it would be really valuable if within like 24 hours of choosing the ttd and the bellatrix height clients are like able to schedule a release um so i and like i'm not sure what's the best way to do that anyways we can discuss it offline but yeah basically we don't want to like have a week go between like choosing this value and announcing it um with client releases so i think that's the only thing it's like it would be neat if clients were like ready to release just before we choose it and then they plug in those two values and then within like the next day there can be a a release that we can

announce

Adrian Sutton: yeah one of the things to be aware of is that until we have a ttd whether it's a ridiculously high one or the real one um users can't start updating their config because the execution engine on a number of els doesn't start listening um and you know the config checks don't happen so if we're making that a tight timeline it it's challenged for users to operate in time

Fabio Di Fabio: we're in the process of changing that so that uh when you start upgath it will already start up the engine api

Adrian Sutton: that's awesome i think the config checks will still be an issue because you want them to be in sync but um yeah definitely useful to change that

okay

Danny Ryan: let us move on other merge related discussion points anything that we want to bring up either to notify this group or ask some questions anything like that

Paul Hauner: i did want to um just touch again on

what marius and adrian were talking about then about just trying to get make sure that all the execution engines will start their rpc ports kind of from now before the merge before the merge params have been announced that allows people to set up their devops environments as if the merge has already happened so yeah it's very important so at the moment eas that don't open their auth rpc port before the merge params it means that people can't future proof their um setups but i think um it sounds like we're people are already moving towards that just want to raise

a disappointment

Danny Ryan: and tim can we make a note just to bring

that up on awkward devs as well

Tim Beiko: yes

Danny Ryan: although i think we have good execution layer attendance today um okay great

any other merge items

Parithosh Jayanthi: do we want to talk about girly today or status update on

girly

Danny Ryan: yes do you have a status update or does tim have some ideas on when we can or should be

picking numbers

Parithosh Jayanthi: so the only thing i have that could help is this website from free um it helps us estimate ttd on gurley because he's really stable to estimate i think even if we were to pick one now that's three or four weeks in the future we'd still be able to get it down to within a few hours um yeah and the website lists like a bunch of options but that would i think depend completely on the client teams and um when they want to try and schedule it for

or even if we want to make a decision today

10

Danny Ryan: based on recent discussions is that something you want to do today

yeah

Tim Beiko: i'd just be curious to hear from client who's like i know on the last core devs last week you know a bunch of them were looking into some recent issues um especially on the el side um and if if we can pick it you know far in the future is is it valuable to pick it today if it's a month out or is it better to like pick it closer um i don't have a strong feeling there but yeah i guess do client teams feel like they're ready to move forward

and if so when would they want to see the

the force happen because because

Łukasz Rozmej: uh so from another mindset i think generally uh it's good to have a plan ahead we can like modify it if something goes terribly wrong and in terms of like the dates four weeks is reasonable freeze a bit stretching it with with the issue we uncovered that we want to explore more um yeah we would like to have this fixed before

girl and that's it

Tim Beiko: you said four weeks would be reasonable sorry

Łukasz Rozmej: yes yes

Tim Beiko: okay so four weeks that would be like august 10th or so uh one two three four yeah if i'm

counting correctly um i guess and then obviously we'd want bellatrix to happen before that um as we're all talking about i think it could actually be interesting to like try and try and like estimate it four weeks out because um that's basically what we'd want to do for maintenance um we can have bellatrix happen closer like we don't need this like equidistant gap like we do on mainnet because of the hashtag but

Danny Ryan: um yeah monday a monday bellatrix might make sense in this type of network

Tim Beiko: yeah exactly so we could do uh basically targets the 10th of august for for gordy and then target the 8th of august for

bellatrix

Parithosh Jayanthi: um i think i'd still keep like few days um more apart just because worst case all the signers then bring up their nodes and they've realized they've been offline and suddenly the difficulty average

difficulty goes down

Danny Ryan: so more of a maybe a thursday prior to that wednesday i'd say like three four days it's probably safe well then you have the weekend

so then you want to buffer the weekend or or we do um thursday on tuesday and monday in tulsa for example right johnny does monday sorry go ahead yeah

Adrian Sutton: the only concern i

have is that if we push out the merge for gauley pryda is the main network where validators are testing um so the later that is and the closer that is to mainnet the less the community less time the community actually has to really prepare and is forced into preparing for this so i worry that we push it back so that we can get the final details of final bug fixes and a gold-plated release that we actually shoot ourselves in the foot because we're kind of we're giving less time for the community testing in the in the biggest environment that we have in the place where most people will be affected

Tim Beiko: how long do you think people should have

to test

Adrian Sutton: um i think i i don't think there's a clear time but just to me it's more let's let's fork goalie and pride earlier rather than later um even if we know that you know a couple of clients have got got some bugs as long as the user experience is set then that preps people really well um otherwise we're twiddling our thumbs and we're not not twiddling our thumbs we're taking up more time towards the merge than we need really and

that's being taken out of a community testing budget basically um and i don't think that's that's reasonable we need to get the community on board as soon as possible

Danny Ryan: so given that we're july 14th right now do you have are you making a recommendation on a particular date or week

Adrian Sutton: um i mean how quickly can we get client releases out basically

Danny Ryan: so in one week we'll have the girly shadow for so i believe we shouldn't do it next week and that we should do a special shout out for before so then if not the tenth we're looking at more like the 27th or the third um it sounds like you'd be pro 27th

Adrian Sutton: yeah i said

um i think i think we need to give a reasonable amount of time between announcement and update as well um because again we've got a lot of people on part of it that will need to actually make changes and they won't be paying attention too much so 27th probably works

Danny Ryan: phil you mentioned the community call tomorrow is um those that have been on and been running these calls is that actually a good place to like get input on this casey i'm

muting you

Tim Beiko: i don't i think we can get some input for sure um probably by like the more professional side and like infrastructure provider like i don't think there's going to be a lot of like

individual random stakers who show up

Danny Ryan: but i also don't suspect that well i would suspect that people want more testing time um so that yeah i would estimate that's the response

Tim Beiko: i think yeah one thing i do agree with adrian though is having like some amount of gap between like announcing and release that's like at least a week and not like a couple days um so it means like you know if we announced it on the 27th then we would fork on the third basically but like that requires clients to be ready on the 27th right um you know and more or less a couple of days but i don't think we want like to announce on

a monday and fork on a thursday for example um if we're expecting like most stickers that actually uh have to take time and update there can fix there

Danny Ryan: right and implicit announcing is we want to have uh releases out so it's like deciding today you wouldn't announce today you would announce in some amount of lead time before the fork when we had releases

Tim Beiko: so i guess yeah it seems like either we would announce

something on the week of the 27th and fork on the week of the third or announce something on the week of the third and fourth on the week of the 10th um like i i doubt that like client teams unless yeah client teams could have a release out next week and then we've worked on the week of like the 27th but the el team was on awkward last week i think that was a bit quick um all right so yeah

Parithosh Jayanthi: i mean i'm also questioning if one weak notice is enough because we ideally also want the community tools to have enough time to update for example

the ethereum on arm guys would have to release their image that would rely on the um cr releases that node would probably have to make a release and we'd probably want a few community videos and guides up and running um i'm sure we can already start working on that stuff with like placeholder releases but just so you guys know there's like there's probably it's probably good to have a higher buffer period than just releases that i would expect that we want to update

Tim Beiko: so in that case if we think we want even more than a week i would i feel like releasing on the week of the 27th and like announcing there and then maybe doing the fork like uh bellatrix on like the monday and then

uh and ttd on on the thursday so like on the 8th and the 11th so that gives like 10 days before bellatrix and like 15 days before uh or a bit more than 10 days actually before apologize mark 11 days before bellatrix and like 15 days before ttd on gordy from the releases um does that seem reasonable to people so it means client releases should be out like on the 27th or 28th ideally so that we can announce this on the 28th or 29th and then um yeah fourth on the 8th in the 11th

Danny Ryan: can you contextualize that in terms of

um a main net timeline

Tim Beiko: um i mean look like uh if we if we fork i guess you know imagine we fork gordy on the 11th of august um everything goes well everyone's super happy um it probably won't happen like you know imagine like yeah we we we see it go well we want to see it like stable for obviously a couple days to a week it means on the 18th of august for all core devs we can probably decide upon like when do we fork main net um and you know imagine that like we have the release releases out to the week

after that um or like yeah so that means that then two week after that that's like early september early september and then like mid late september we would hit ttd um so i think it it's as well like yeah if we have the release

Marius van der Wijden: if we have the releases out early september then i don't think we can with

it no

Tim Beiko: sorry i meant early september you get bellatrix so it's like we work on the 11th on the 18th we decide we decide on like the final values so the releases are out like you know like the week of the 24th or so the week of the 22nd of august and then two weeks after that you have bellatrix and then two weeks after that you have ttd um

which which gives you like the week of september 19th um and if we wanted to be like slightly more aggressive you know like i guess the the thing yeah the thing that would maybe buy us another week in a way is like either we don't fork gordy on the 11th we have like a lower gap between bellatrix and ttd or if we want to get a couple days as well we can also have like an off schedule call because we're like waiting basically a week between gordy fork and the next alcor devs to choose the ttd if we want to save a couple more days we could have like an off schedule call or just move all core devs the next week to like the tuesday for example and then

um and then you gain that much that that many more days um so but yeah i guess i guess that's it yeah the only risk is if we fork if we try to hit ttd on a tuesday the obviously the cl call happens on a tuesday and so it's unlikely that on that day we feel comfortable like moving the main net because we won't we may not have seen ttd on gordy hit and even if we have it'll be at best for a couple hours

all right

Danny Ryan: i definitely think we're in a position we're picking up two days or one week here and there if

possible is preferable

Tim Beiko: yeah i agree i'm i'm fine moving awkward abs earlier that week if it means like we get the merch like a week earlier um because if you imagine you know if you had all cordettes on like the even like the monday rather than the thursday it means for sure client teams can have a release out by then um by the end of that week and you you kind of save like you know instead of having bellatrix hit on the week of september 5th maybe it hits on the week of like uh the last week of august and then similarly instead of hitting uh ttd on the week if the the 12th and the 19th

maybe you hit it on the week of the 12th um so i think we can you know and we don't have to decide obviously this today but i think if we if if we're willing to um have uh the gordy fork happen in like the week of the eighth um both the bellatrix and ttd have clients out on the week of the 25th it does put us in a space where it's like quite realistic to emerge in september assuming again that like the gordy fork doesn't completely people up

Danny Ryan: okay so based off of this conversation obviously the conversation got anchored again onto that

8 10 week but there's also more context around potential timelines and how to strategize there adrian or anyone else do you want to still argue for an earlier week

Adrian Sutton: i mean i'd i'd still probably prefer to get client releases out next week with the goalie ttd but um that is probably pushing it um so i think i can live with that i think it's a no-brainer that we should schedule awkward devs shortly after golly goes through and merges so that we can make that decision and call on maintenance as soon as possible

Danny Ryan: yeah agreed thank you anyone else okay

um so we'll have our statisticians go to the drawing board and select some uh potential on the ttd and fork epoch for powder or for gorilla and powder respectively and then tim is that something that we should try to get consensus on over the next few days or is it something to get consensus on on one week from today

Tim Beiko: um i'll share some values i'll share some values like before the next awkward ev um we just need to like tweak a bit a free state and but not with like you know we can confirm them on accords but there's no reason for people to

block to be blocked on that like um i don't know if i'll get them today or tomorrow but at the latest on monday people should expect like a proposal for uh um you know the epoch is like a no-brainer to figure out obviously um but yeah the ttd we just need to change after his estimates to do thursdays instead of wednesday so that shouldn't be rocket science

Danny Ryan: okay thank you

Adrian Sutton: um any other to decide that eight seconds if we can instead of waiting for awkward to set a ttd otherwise we just take another week before we even start all these processes

Tim Beiko: um thank you so i can

i can put um i can put it in like the the spec for shanghai and i'll open a pr and i'll leave the pr open for like 24 hours and somebody wants to object to it and they can there but then i'll just i'll just merge it

Danny Ryan: okay thank you any other uh points on this girly and uh merge schedule okay any other discussion points related to the merge

okay thank you

moving on mv mev boost uh metachris can you give us a quick hey everyone yeah

Chris Hager: sure um please interrupt me as a couple of points and i try to keep quick one notable development is that prison finished the beacon specs the builder specs implementation so it's now taco and prism that are completed integration and it's merged i think lighthouse still has a testable branch but it's not merged and nimbles and loudstar are still in development next topic is that mapboost had a

security audit that is now completed a bunch of minor issues with some recommendations and one medium issue that is uh important to put in some specification for relay operators there yeah that's like some edge case i linked a github issue with the audit pdf releasing a new version with all the minor fixes very soon we've been rolling out updates for the flashport's release and builders across kiln robson and the polio so we're seeing a lot improved performance

thanks everyone for your latency measurements and reports in particular enrico and terence it really helps and let us know if you see any other issues with the performance the bottleneck is mostly the signature verifications of the registrations because on the beginning of an epoch we get uh 10 20 000 registrations and prism is currently resigning the registration on every epoch so it needs to be pre-verified yeah so that should all be i'll already done we got a request from booster to make one of the test nets

open to any validator so registrations and getting the payloads are not bound to actual validators i think we can accommodate that easier than the polio what else uh good to mention this is a permission network

Parithosh Jayanthi: so it would kind of defeat the purpose it might be better to do that on drops

what is it

Chris Hager: really only needed for consensus client development not for actual users

ah okay okay make sense thanks for the

good idea

Chris Hager: so yeah the real is they are in a good state uh if you see any additional latency issues we might uh go to investigate the network crosses because the relays are fast now from our end there's an now an ongoing push on specifying the proposer config file format i know enrico terrence and and stefan have been working on that and now the push to make put it in writing which is a config file definition which allows proposers to choose relays fee recipients and other properties

independently so basically available clients with a bunch of elevators where each validator can set custom properties and then fall back on default properties

maybe i should link the

Danny Ryan: yeah so akin to how graffiti was done so you can even programmatically change the text file if you want

Chris Hager: yes uh and yeah and lastly i wanted to give a shout out to the ef security people that have been looking into my

boost builder utils and other repositories in particular shout outs to justin trudeau he is a very was very good in finding like edge cases around the ssc encodings and some types and yeah that really is very much appreciated it's mostly around the go boost utils repository which is like just that all the types for the builder specs and the ssc encodings and json encoding and decoding functions all right i think that's it around the map boost most important status updates

Danny Ryan: great i have a quick question the audit

that you did was that on specification and implementation or just the implementation

Chris Hager: mostly on the implementation but he also looked through the specification but it was specifically on the implementation

good

Chris Hager: i think he found an edge case that is um not necessarily something to change in the specification but rather on like nodes for implementers where validators could if they really does not properly verify designed blinded beacon block corresponding to the real slot and proposer then there might be a way to to get the

payload even if you're spoofing the proposer like you can read it in the pdf linked the medium issue on our release this is fixed but we should put this probably in the builder specs repository in a relay document

got it i will make a proposal soon

Danny Ryan: okay any other questions for chris

Parithosh Jayanthi: yeah would it be possible to have aurelia up in a couple of weeks for one of the shadowfox we could have something like 10 20 of the network blank and so on on mev boost and see how it looks on a wider scale

yeah

Chris Hager: we can do that for sure i think we just uh need to rely on the timeline uh we just need a little bit of a heads up to sync up some notes and and test this

Parithosh Jayanthi: um yeah if we have early shadow for next week then i guess we would do midnight shadow for the week after so that would be about two weeks in the future

Chris Hager: okay yeah i think we should be able to accommodate that that is the end of july king of august that sounds sounds great

Parithosh Jayanthi: okay and um in case someone has opinions on how much of the network has to be any mev boost feel free to reach out to me and then we can make that

happen okay anything else for chris

Danny Ryan: great thank you chris and alex i believe brought this up on all core devs last week but there is an issue or a pr up in the build respects around murder transition delay alex are there any remaining questions or comments you have for this group today

Alex Stokes: yeah i think uh mainly we just wanted to talk about implementation i think some people here on the call uh wanted to discuss that as expect right now it would essentially be like an epoch based delay uh so the first question is simply like does everyone think this is a good way to you know sort of specify this embargo

Enrico Del Fante: it was it was me um

discussing on the on the pr issue so um the delay was initially thinking i was i was thinking that it was was been has been a very easy thing to to implement but the way this been specified it was adding a little bit of additional complexity than more than expected because actually if you need to add a delay based on when the first uh execution payload has been finalized if you start up a node and the node only has the latest state you don't know actually when uh in the past this execution payload has been has been first

has been first uh finalized so you cannot count and you cannot uh bring up with the with this delay correctly unless you are persisting this information at runtime when it happens and then you use it later this was my my point and it's definitely something we we can do when we were at we were arguing if uh makes sense and then the argument from alex was actually convincing that makes sense for our odyssey our clients to implement this ei as well but i'm thinking if there are other clients thinking about that

is

Paul Hauner: the challenge in knowing whether the finalized block is post merge transition or not is that a challenge

Enrico Del Fante: yeah actually because at least we have the only the latest finalized state right so image that you start up the node and you have the your latest finalized state with the execution payload already there so the chance is that you have to go back in time with the canonical chain finding out the block that was bringing the the first execution payload in the past unless you have you have stored somewhere when it happened in the past this feels like

you're

gone i'm gonna

Paul Hauner: say at least for lighthouse we so in in fork choice we track um whether the execution status is enabled or not and i would assume that most clients are doing it because of optimistic sync um and i don't think you need to actually find when the transition happened all you need to know is whether or not the finalized block it is pre or post transition i'm just thinking generally as well i haven't read that

spec sorry

Danny Ryan: yeah i mean there's also just the practicality of this is something of an edge case of somebody who is syncing with presumably a

finalized state that's after the merge um and if it's i i would imagine that when you're in the hundred epoch range or something like that that you can kind of just turn this on um or if your node is online for 16 epochs then you also know that you've now reached the condition because you started with a finalized state that had the execution payload in it and now you've also weighted 16 epochs so you know that at that point you've also hit it so i just i think we can probably use some sort of heuristics here for this edge case around sync if somebody's actually starting with a a piece of um

a state that's really close to the merge and there's some uncertainty but i wouldn't i don't think that we need to bring a ton of complexity in

here is

Potuz: my intuition is the question about this uh the check for optimistic sync whether or not we're close to the

port

Danny Ryan: the question is if you're if you're actually at plus 16 epochs since the first finalized post merge state

Potuz: i'm not understanding why would you want to add the 16

ebooks

Danny Ryan: 16 is the suggested number in the builder specs on how long to delay turning on

muv boost

Potuz: oh i'm sorry this is okay

Terence Tsao: yeah so for prison it's actually fairly

easy to do since we can just look out for choice store and see the latest finalized blog has a execution pillar or not and then we can plus minus 16 epoch what of slot from there

Adrian Sutton: but if you know then restarted you like so if you then finalize your four epochs into that period the note restarts you don't want to add 16 to the finalized checkpoint when you start up again you've actually got to remember that your before epochs into that period which there isn't anything in the state that can tell you that i don't think

or broader right oh yeah

Paul Hauner: i didn't realize that we were waiting like any epochs after the finalization happened yeah that's going to be a little bit more difficult for us as well

Enrico Del Fante: yeah i was thinking if you are at runtime you can do that if happens at runtime you can we can store something in our key value store just for remembering and persisting for for the sake of our start and then and then use it uh if if the number starts it was my my intention to lower down the complexity even if at some point you need to hit the store and to persist this information which i think it was not intended in the first

place when writing this this back this was my intention

Paul Hauner: um i don't know if i'm just recovering things that have been discussed already but how attached are we to that extra and epochs past finalization i would be tempted to argue that if we managed to finalize the chain then it's time then we were kind of out of danger zone in terms of a bunch of weird edge cases with merge and that it's okay just to turn mev on then

Danny Ryan: well you could also do plus and epochs from point of merge and finalization or is that hard to pull up to

Paul Hauner: having just debate but just based on finalization is super easy um and apocs is harder

Danny Ryan: i think from point of merge not just finalize merge

Paul Hauner: yeah i think so because finding the point of the merge is the tricky thing yeah we are already sorry

aren't

Mikhail Kalinin: clients download the blog historical blogs i mean like going back beyond the checkpoint that they've got on the startup because 16 epochs is like uh yeah it's just 500 of blocks so you may

just pull them from the network if it if it makes sense i mean if it doesn't if it's used for some other things yeah just look at the block yeah that boundary

we

Paul Hauner: we do backfill but the the the complexity isn't then saying like oh no hold up don't produce a block until you backfill that's not something that we have and it's something that i'd be hesitant to put in the flow of block production

Mikhail Kalinin: not do not produce but just do not use map boost i don't know probably it's complete it's just complicated

Paul Hauner: yeah i think it's complicated for us for us like backfill is like this thing that we kind of put in a thread and shove over to the side because it's very um independent so it'd

be nice not to link it all through to everything

Mikhail Kalinin: um also there are historical blocks in the state yeah but i guess for magnetic can work so you can just pull the block by hash the exact block and just look at this uh block

Ben Edgington: it seems a little bit missing the point uh to yeah yeah kind of complex solutions because the whole idea is to make things simpler for ourselves around this

Danny Ryan: yeah i agreed i'm if this requires increasing complexity that's bad um so i i would i would error towards

simpler and maybe just uh paul's suggestion

Enrico Del Fante: yeah i mean i have just one very easy easy thing is to simply do a delay not not based on the on the on the finalization but just adding a a delay on the on the bellatrix fork which is that will be easier but you then you need to find out the right value that ensure you that you go after the ttd and though to be sure then it means that the mvv boost will be activated white later than expected

Danny Ryan: yeah i wouldn't

i think that that pushes us too far in the other direction

Mikhail Kalinin: yeah timing i think that there will there will not be like a huge portion of validators who has just recently joined uh post merge which means that this parameter and this trick works is like more for those who are online and staying online through the merge so probably if you're starting new a node from a checkpoint and it has a payload that is finalized it's okay to use map boost right from this right from the beginning i mean like when the node is ready

what was that was whether the relayers

were going to be playing ball with what there aren't that many relayers and we can just start map boost on our end since we are finalized and just ask the relayers of jane just don't turn on the relay or don't accept anything until 16 epochs after finalization then

Mikhail Kalinin: there is a risk of uh another layer that will just not agree to follow this item

i'm sure

Potuz: yeah but it's very minor because anyways we're analyzing just running this after finalization

Mikhail Kalinin: yeah

and i think if your cl is starting up with the checkpoints probably probably your el will be sinking like for the next few hours or whatever i don't know so it will also take time where the node will up and ready for producing blocks so i would i yeah my opinion is just yeah do it in a simple way and if if checkpoint has a finalized you have to quite already have has the payload in it so just start using map boost okay yeah i'm i'm in favor of the finalized thing i think doing it inside

the beacon node is good the beacon node has the finalized state um it's the ultimate controller whether movie boost gets used or not so doing it there makes sense um and doing it based only on the finalized state um it's something that we know clients always have it's simple and we don't need to add edge cases about did you recently sync or not

i

Enrico Del Fante: i do agree we're already doing that and having a finalized execution payload in any case gives you gives you a good hint that network is healthy

Danny Ryan: okay so seems like there's general consensus on simplifying likely simplifying so that the consensus layer just uh those clients can begin

acting will begin acting on amiibo boost at finalization there's an additional thing where maybe it's specified that relays relate weight in epochs because they don't have the same edge case concerns around like syncing and re-syncing that kind of stuff um is that a reasonable synopsis and can we take that to this issue for further discussion

okay

Alex Stokes: yeah i can update the tr and that sounds like everyone's on the same page there

Danny Ryan: got it okay

anything else on this one okay we have 15 minutes um are there any pressing updates any pressing research spec or other discussion points for today okay any non-pressing updates or discussion points great

okay thank you uh good meeting and we will be talking about gorilla configs over the next handful of days and keep the ball rolling appreciate it talk to y'all soon

bye thank you [Music] so

[Music] so [Music] you

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.