All Core Devs - Execution #137

Transcript

[Music] non-discrimination [Music] [Music] thank you

[Music]

Micah Zoltu: that's some foreshadowing right there

wow

Tim Beiko: so right where we searched three might go into comments welcome everyone to awkward devs uh number 137 today so it posted the agenda in the chat uh we have a bunch of of merge related updates um Frankie that's probably all we're gonna have the time for um

yeah and yeah I I guess you know to kick us off uh do we have Kerry here yes okay so we have Perry um Perry do you want to walk us through like the two Shadow Forks that happened last week and uh where yeah what what happened there but um so we had two Shadow Fox last week when um during their connect the first one was a girly Shadow for girly Shadow Fox 4 and this was I think the first Shadow Fork where we had multiple clients taking on and I think everyone made it through the transition but bezu and Aragon post transition had issues and stopped

working uh but during the week the teams pushed a bunch of fixes and we had midnight Shadow Fork 2 on Saturday so that's about six days ago um and minute Shadow Fox 2 was like it worked a lot better we didn't have any major issues it seemed that all clients didn't hold through the transition and also well after we did uncover a couple of issues with deposit processing and we were looking at more ways on how we can Harden that um we did have an issue with late blocks uh being proposed by Prism it was a fixed pushed like relatively soon after it was discovered and the net has been quite good since then

um the other issues we found through the week was some proposal related incompatibility between Nimbus never mind that's been fixed now we had one I think we had two issues with basal prism um but I think that's also been fixed right now and eragonism is still um undergoing triage I don't think we know what's going on there yet but there are other Eragon notes that are in sync so it could just be some some incompatibility we have to figure out um but in general the network is stable I think we're looking at like 96 ish percent participation and we're hunting down the rest

awesome thanks for sharing anyone from any of the client teams want to add some some comment there

Andrew Ashikhmin: um yes I'd like to add that occasionally I I hear reports of Aragon nodes being stuck especially when people try to think uh main action of work some sometime afterwards so I have to investigate these things stock issue and also like fixing Hive tests it's something on my plate so still a lot of things to fix in Aragon for the merge

call it uh any other crime team

Péter Szilágyi: yes uh I have one I've been discussing

with her it's a bit with diary after the shadow Fork and I got this also ties into this uh Ariana before that um as so these shadow chords are nice to test the transition they are always testing it with perfect clients so to say namely that other clients both the dica clients and the exhibition clients are in sync so everybody's just going in lockstep and waiting for the thing to hit and um I mean whilst that is nice I think it will be also interesting to somehow create some tests where certain kind combinations uh are out of sync or out of wax essentially and uh I probably

setting this up would be quite messy or quite complicated maybe it's an API of points would be needed to actually test these scenarios it would be super nice if we could have for example some nodes that are let's say the beaker client is in sync but the attribution Hyatt is just thinking whether that's initial snapping or just behind a few blocks and doing full sync and it would be nice to have the other way around where the execution client is actually following the ptd but the works are following the chain progression but the beacon client is out of sync and the reason I'm kind of thing is that I assume that people will try to be sync around the

around the merge and it's going to I mean it would be nice to at least confirm that it kind of works okay even if there are some works here and there

Tim Beiko: is there yeah I can imagine how setting that up automatically would be tricky but is there a way that we can maybe have client teams like manually test that on one of their Shadow Forks

Péter Szilágyi: um I guess the problem here is how do you um probably that's not really reproducible so if nothing goes wrong you have no idea what happened

and uh and then also at least speaking from God's perspective we have a API call descent head so what we could do is right before ADB hits maybe just set the head back I don't know 60 blocks or something and see what happens um I don't know if something like that is available on on bigger clients but I think it would be nice to be able to sum even if not a very exhaustive list of corner case checking at least some some basics because the problem is that um so what we did test with this Shadow Force is that the whole thing can just transition through it and that the conditioning is just the beacon

client is feeding the execution clients blocks one by one but if let's say your node is offline for for whatever reason you just install Gap because it wasn't ready for the merge and you restart it and all of a sudden and nothing works because there's some synchronization issue then it's getting it's going to get messy because eventually all nodes will it will happen that you miss a block for whatever reason and then you need to actually fall back the proper sting and that needs to somehow tested

Tim Beiko: got it um anyone on the testing side have any comments

about

Tim Beiko: I think we

[Music] oh okay so Maris has done some manual tests for stopping and resync can get notes you want to take a minute maybe to walk through what what you did do

Tim Beiko: you have audio oh yeah yeah

Marius van der Wijden: sorry um I might have caught something in Amsterdam um I did a bunch of tests where I stopped the beacon notes so they get the consensus at the execution layers would

just stand there and not be fed any new blocks and then restarted the beacon notes so that the beacon notes sync up and feed us the blocks uh one by one I also wanted to do a bunch of tests where we delete one one or the other databases so I have a sync node delete the contact layer database resync the consensus layer with the execution layer running and the other way around

um [Music]

Marius van der Wijden: but those yeah I haven't done those yet

and it's not that easy to um well you can like pretty easily uh uh write a script for it um but yeah I think it needs some manual like looking at to see if it actually works

that's it oh I also did a bunch of set head tests uh on the main it should have II um where I said head a couple blocks prior also a couple blocks before the merge so once the match was happened I set it on the get node before the merge and that went

single time awesome um and

Tim Beiko: Peter just to make sure I I understand exactly what you're saying is like you want to make sure that we can have kind of these imperfect clients with like

Alice saying

Tim Beiko: States when the merge is happening right like the fear is not that like after the merge they can't get into sync but it's like as we're actually going from TTD to finalizing on the other side that's when we we want to at least see what happens with with clients is that right

well I guess both yeah during a transition is uh so the transition so I don't know how other clients handle the transition code but in that there's quite a lot going on

Péter Szilágyi: because there's a up until the point you you can just use Legacy sync to sync up until the DVD than if you reach the TTD then you need to wait for a beacon client to to pop up if there's nobody about listening this is a bit different for full sync and snap sync so there are these weird scenarios that we try I mean I invested in quite a lot on the various I think on the kill desktop so we do try to extensively test it

I'm

Péter Szilágyi: just saying that I don't know if other clients did some did tests around these issues and it might be worthwhile to make sure that it's that they are kind of okay with these hiccups

Tim Beiko: right yeah I'm curious to hear from

other teams if anyone has anything they want to share

Marek Moraczyński: so for us it's like we are still a bit working on these cases uh and we are aware of some problems so um we need to test it on the killing and Shadow Forks

Tim Beiko: got it anyone else okay um and then I'm curiously how

um is

Tim Beiko: is it a possibility to also kind of tell people that they should have a sync node when the actual merge happens and if not basically sync it on the other side or something like that like where um yeah if you know like I think if they're in in Step In Sync like pre-merge they should obviously be fine but then and and then if they think once the merges happen they should obviously be fine there as well but like um yeah it's is it like possible to just tell people like I mean yeah in in an announcement blog posts and stuff

we would definitely recommend people are synced uh but yeah you know uh they're someone will be thinking during the merge right yes yes and but I assume there's a difference if it's like one percent the valid letters or 35 right yeah absolutely I mean we how would we generally assume that most validators are are in sync and if you had 35 out of sync you wouldn't finalize which you know is fine uh but then they'd have to get up the head and

think I want to just uh sorry

Mikhail Kalinin: yeah I just wanted to Echo Danny what

Danny has just said that those who are seeing him probably not that important during the transition what's important what's more important is that clients will be able to sync up uh after the transition is finalized like from scratch or from like different

areas

Martin Holst Swende: one thing um that might be worth highlighting so previously it was fully possible to sync up your guest node or another my node without touching your validator and without them communicating post merge it's going to be more problematic right what do you mean

I mean that we don't see the head so for example if you try to start yes without the validator yes will yeah without the beacon note is what you mean yeah yeah it cannot reasonably sink to the Head

it

Martin Holst Swende: I don't know if it would try to latch on to the latest Google work block or something but then it wouldn't be a possible to obtain a state from anywhere

I mean

Péter Szilágyi: that is what it does so it will try to sync up until the DTD enforcing because it will just stop at the DTD and just print the warning

there I mean I'm talking about I mean people don't use postings yeah I mean Snapseed

um it will try

Péter Szilágyi: to I mean the chain itself will be sync up on your TTD and we try to stop and for the state it will just try to sync the head which will be the latest head will be DVD or something so that will go a bit screwy so if you try to stink at that point that somebody actually feeds you some stale data because they are also stuck on the

TPB

Péter Szilágyi: then you will end up thinking to some weird States and at which point healing even if you attach a beacon client healing will take forever because we'll just go back to the classical

passing algorithm yeah so screwy situation

Martin Holst Swende: yeah and you might actually kind of wind up in a worse situation than if you just

had not sick because then you wouldn't be talking about over over snatching so we should whenever we issue some documentation about this the thinking and merge we try to highlight yeah

Martin Holst Swende: because I guess the same thing needs to be true for other clients as well and presumably that

yeah needs to run with

Tim Beiko: uh yeah and the kid on blog post we already highlighted that like basically if you run a node post merge you are running two pieces of software right like the consensus layer and the execution layer um and I think we're gonna yeah in the

first like testnet announcement post we're going to try and even expand that more uh to you know uh yeah just for stakers for non-stakers uh for people who are running a node on on the execution layer today um so that it's very clear what what they need to do um because yeah if you're if you're running kind of just yet without a consensus layer post merged and you're kind of not running the full ethereum chain yeah there's also the ability to service warnings and errors to the user because exchange transition configuration would not be happening so you wouldn't be getting any um ping essentially on the engine API um that also yeah I mean you could do a lot of that

information but at the bare minimum you could expose some big wiring to users

Micah Zoltu: I feel like it should be more than a warning it feels like the client should not try to sync in that scenario like if if the execution client knows that we're supposed to merge it should just say hey I don't have a vegan client I can't sync oh yeah but yes you slowly might not know it's post merge it might just know that if it's been released as a merge client well but it knows the ttd's been hit right because it's going to talk to the network and see that it sees TTD blocks and then it'll know hey I know that we're at least at ctd like uh maybe we're at

TT exactly but we did not verify that they did be until

you actually reached the point so I can just feed you hey here's the DTD we already pass it and then you will just get stuck because you think the network much

Micah Zoltu: so that's for if you're doing a traditional sink if you're doing like a snap sync how does that work

exactly the same way you need to validate it 150 to the otherwise [Music] but

Martin Holst Swende: I mean the books and the headers up to DTD those are still valuable so that's that's kind of fine

in

Péter Szilágyi: so one two

one thing I wanted to propose I we have it open as a PR in gap for a month now we just never rush it because I want to talk it over with a hcv is that and we've been discussing that after the merge clients would release uh it would make a new release in which it is actually marked at this network practitioned and we just come field in the in the Genesis

pack

Péter Szilágyi: and that would be super helpful exactly to circumvent this scenario from causing couples long term because we can say that okay

there was one field which was a dpb but we can also add a second field I don't know total terminal difficulty which a

Boolean or whatever

Péter Szilágyi: which actually signals that the network has already successfully merged and from that went onward the execution clientele can very well add Genesis block say yes I've seen blocks in the network I'm not going to touch anything until I have a bigger client that's giving me something meaningful and I think that's a long-term solution

Micah Zoltu: is there a reason we can't include that in the um the the before merge release like is there a reason that people should be able to run a um proof of like the latest release that we you put out right before the

emerge

without aving clients this flag essentially disables like a system you would not be able you would refuse processing blocks from the network but maybe Mika

Tim Beiko: uh one thing is there might be a world where like some peers don't upgrade their clients like pre-merge and we just lose them so I don't know how much it affects things she doesn't yeah

Micah Zoltu: I assume that we'd still be able to communicate then like you'd still be able to communicate with people who haven't upgraded up until the merge

Tim Beiko: right

Micah Zoltu: right so emerge actually happens to you lose communication yeah and so if if you're running an emerge client and the merge has not happened yet it feels like and I'm sure I'm missing

something here but it feels like we should just say hey if you don't have inclined at that point your your execution time will start like it feels like I can't think of a good reason why we would want exclusion clients to start up without a consensus client paired with them once they're running emerge clients I I do think that that's not totally unreasonable I mean there's two options here one is you do a huge warning the other as you say this looks bad you need to actually make sure this other thing synced and I'm not going to do anything until you do so right

Péter Szilágyi: that seems like a very surprising way to lose half of the network that nobody accepts it they need to have a vegan time up until the

merge and then all of a sudden a month earlier how the network stops well this is only people that have been that have run the like guest that is released for the merge and so they're either going to fall off if they don't run the other side they're going to fall off a month before the merge and hopefully fix it or they're

going to fall off at the merge communication Works whereas if I

Péter Szilágyi: add this featuring and all of a sudden it's not about making sure you have a correct setup at the merge rather you need to make sure that you have a correct setup at exactly the for the upgrade of gas but since you haven't upgraded yet you don't even know what the current setup is yet

you're concerned

Micah Zoltu: here if I understand correctly is that a user who upgrades their their guests you want them to be able to upgrade and have downtime of like you know a minute however long it takes them to upgrade and then they can iterate on connecting to a dating client yada yada and they have a month to do that whereas if yes refused to start without a vegan client then when they go to upgrade they're offline for you know a week while they figure out how do I run a consensus client

yeah that would be my main concern with refusing the stock prior too much

Tim Beiko: wouldn't that be kind of good though because like I mean they can still run the old version of Geth you know like they're offline in that like they're not online on the new

version but isn't it good if we have clients who are not like all messed up on the network because they struggle to set it up and if they're instead kind of forced to to do that and stay on the old version until they they figure it out

Micah Zoltu: I think you want to be careful that the execution client does not do any sort of like database upgrades or anything like that internally um until after his established communication properly so you can downgrade basically makes if you want to do that you have to make sure that the downgrade path is very very clean like this people will you know upgrade to the latest version and then find oh I needed a consensus client to make this work and so then they go to downgrade you want to

make sure that downgrade works very very well

I don't know the

Micah Zoltu: individual clients that may be very easy or maybe hard depending on design I don't know

so one issue that I kind of yeah depends on how you set up

Péter Szilágyi: I don't think it's always super clear as what the dollar rate path is for example if you're installing via you go to ppas and there's essentially no downgrade that you just it is just up so that might be a problem here with Docker you know if you're just using you're just putting the latest table then again you somehow need to start poking around your deploy scripts which may or may not be a problem

again I

Péter Szilágyi: ideally if you're running a production setup it shouldn't be a problem but hey

I know it's a

Péter Szilágyi: plus the other potential question is that how do you even tell the user that get is out I mean that you need to do something for example if I just you get refuses to start maybe it will just go into some book Loop where some kubernetes manager keeps trying to start it and we keep refusing to start so somebody needs to somehow dig up a log

and

Péter Szilágyi: where you have a downtime of I don't know half an hour until since I've been

actually figures out what's wrong it I don't know it's just seen that a lot of things can go wrong

Sam Wilson: I don't think many people automatically upgrade gath so I think if somebody note like is manually upgrading Geth then they're gonna notice if it isn't starting right away because the counter to that is if it starts and it just works and they don't look at the lugs which are warning them that they're vegan nodes not synced then they're gonna fail at the merge um but again I think that this can be left to clients it seems like a user you know user experience decision that each client can make

Tim Beiko: yeah and and on the communication side like we will kind of communicate loudly and already have started that you need

to run both parts um I think this is probably the most anticipated upgrade in ethereum like no one who's running like uh infrastructure level production of nodes like does not know that the merge is happening um so yeah I think as long as like it's it's carry explained like in clients like what what the behavior is and that we clearly explain in the announcements like what is required of different stakeholders and like how you know how should they set up their infrastructure um yeah it seems unlikely that like you know most people are a large part of

infrastructure would do this wrong like there will be someone somewhere who messes this up um like we see in every single Network upgrade but I think you know the vast majority should be very well aware that this is happening and as long as we have good Communications they should be able to uh to set up um and uh it's not true there were a bunch of miners who messed up London um yeah I I spent a couple days after London reaching out to the people who had messed it up so there will be some uh but that's that's expected every time

I'm I guess

yeah oh sorry Peter are you gonna say something

Péter Szilágyi: so I know to me it seems that this approach is opening up with a bit of kind of forums I mean every client is has their own cross to there but um one issue that I can see it is that let's suppose you do have a proper setup already you do have the Deacon height you do have

yeah

Péter Szilágyi: properly upgraded and then you just want to restart your system and

well your yep know this

Péter Szilágyi: started up faster than the beacon client and Google me just says if you're refusing to start because there's no Beacon high but yeah because it's just starting or what happens if the beacon client just drops off because you update or restart it now

I think it can be I mean depends on how how aggressive this mechanism is but you can end up with brilliant scenarios

even after the March for example but if that anyway so there are

Péter Szilágyi: I think it's a bit dangerous to refuse the star just because something that should be there isn't yet there

got it um and

Tim Beiko: and yeah earlier on in the Chad Perry said he can try and manually set up some of these like out of sync instants so we can actually um you know run a bunch of of manual tests on it at least

um yeah I I don't know beyond that is there anything else people feel we we should be doing and obviously having all the client teams kind of test their own software um you know in in the various combinations of of other other clients and and making sure that they they walk through the edge cases but yeah beyond that I'm I'm not sure what else we we can do

Marius van der Wijden: so in in Amsterdam we discussed a lot

of

Marius van der Wijden: uh about some testing tools that we need or that we could Implement uh and um Cena from the guest team has already

started implementing some of them in gath but it might be really really good for other client teams to also Implement some of the RPC calls so that we can at least make uh debugging issues if they may arise way easier

Tim Beiko: is there yeah that sounds really reasonable is there a PR or like a spec somewhere where's of

what Cena's implemented

Marius van der Wijden: uh I think we have an issue to collect all of these things I'm going to find it

and drink it cool

Tim Beiko: thanks any other thoughts comments on this

any other thoughts on uh just the shadow Forks in in general okay um and there is another shadowfork Perry planned for this Thursday that's correct yep exactly next Shadow folk is thirsty um my nodes are already sinking I just have to upload the configs to GitHub and I just share a link soon okay

um sorry is that minute should have hooked yeah that's mainly Shadow folk okay good

Marius van der Wijden: actually that's a good question does

anyone see value in curly Shadow Fox anymore or do we just keep the last couple around and um I think if we were to like if we were to automate the process somewhere uh like a bit more then we could have early Shadow Fox every like two days or three days

I

Marius van der Wijden: I would I would see value in that otherwise if we're going to only manually do two shadowfox I don't see see the value I'm doing the Incarnation of folks anymore okay sounds good um and another thing is I'd like to

duplicate midnight Shadow Fork one and we'd keep Main and Shadow Fork two around so the one that happened last week um unless someone's testing anything on one if not I'll let it applicated later today

Tim Beiko: okay no objections

um anything else on Shadow Forks generally okay

Marius van der Wijden: oh uh yeah yeah because Jamie just just wrote

um we had we saw one issue that we only saw on the curly Shadow Fork so um yeah I have to think about if if it might make sense to have another one or two girly folks was there what was particular about Gourley that made this thing happen

probably yeah maybe some weird snap packages

Tim Beiko: okay so if we can automate them we definitely should try and have them running regularly

um and then yeah we can do the main net one this Thursday last call for shadowforks um actually one one more point on the automation front um when we were talking in Amsterdam we so the automation for shadow Fox could easily fit into kurtosis um but when we were talking about it in Amsterdam we said that we'd rather use the dev time allocated to kurtosis for testing like weird edge cases for example pausing Docker containers around transition or um I'm not sure we have to figure out

what other weird cases we want to toss in there um do we still want to go down that route or do we want to focus Dev efforts on having early Shadow fork in ketosis I don't think we have enough manpower to do both

Marius van der Wijden: um I think the the weird stuff is more important than than having just uh like more often Shadow folks sounds sounds good maybe someone else could take over

and and do it in parallel I don't know if we can onboard someone into contosis um yeah those scheme itself is willing to help us with this but they just have enough manpower for one of the two I don't think they have enough for both um but yeah maybe someone from our side can do the other one and I'd also be more in the side of having weird edge cases and ketosis mainly because we're doing so many Shadow folks regularly that it might not bring us too much to have it in the CI

cool and

Tim Beiko: yeah I mean structured

weirdness you know the shadow Forks live we're supposed to try to find weirdness but anytime we can structure and make sure we hit the weirdness over again it's very good

yeah foreign

Tim Beiko: okay um next thing we had uh on the last call we discussed kind of the latest valid hash issues uh Mikhail I know there's been like a lot of conversations uh about that over the past two weeks do you want to just give us a quick recap of where we ended with that

Mikhail Kalinin: yeah and Amsterdam had decided that the engine apis specs stays the same and

your client will just adhere to this pack but also we'll cover this with the tests so yeah basically that's it

um yeah

Mikhail Kalinin: it means that El will respond uh with the most recent valid ancestry hash in case it's invalid block is found on the Chain which El is syncing with and cl may use this information to remove invalid sub-chain from its block tree

so that's basically

Tim Beiko: okay got it um and oh I guess first off anyone have thoughts comments on that

or

Justin Florentine: um yeah hey this is ASU here we're we're

currently looking at our latest valid ancestor code and it it looks pretty much doable for what Mikhail is talking about but I could personally use a better definition of what the latest valid um block is real quick um our understanding right now is that it is the uh common ancestor um that has been validated as it has been considered valid is that the simplest definition we've come up with yeah I mean if you if you you have you have some block you're validating it's the earliest Block in that chain right defined by that block that is valid got it all right um I think we're clearing it thank you

Tim Beiko: anyone else have questions comments about that okay uh next up Mikhail this morning you're also posted another kind of request for comments about uh an engine API response status do you want to could you go over

that

Mikhail Kalinin: oh yeah uh this is the real estate as well attached and this is a kind of blind spot in this deck currently um so the problem is that if literally the first proof of stake lock and the chain

isn't valid uh what

Mikhail Kalinin: what else should return as a way to spell attach

uh it it may return like the proof of work block that is the current uh of this um but this information isn't relevant for cl it doesn't have this proof of work lock which is basically the terminal for Block uh in block three so this request performance about let's take a look into this um we may do we do nothing with that but see I will have to um stop traversing their block 3 if they found that there is an empty execution payload in the blog so it would just mean that this literally transition

block is invalidated and that's it or we can be more uh specific about it and more explicit about it to avoid some edge cases potential which creates that so just take a look and comment on this issue

yeah this this is like an engagement for El and client Developers this is

Mikhail Kalinin: yeah it's kind of in the middle of

cool

Tim Beiko: yeah I I think he posted this literally yeah five hours ago so I don't know if anyone's reviewed it already but if so

feel free to comment here

Mikhail Kalinin: uh yeah

Justin Florentine: uh basically reviewed it we didn't have any concerns okay

Tim Beiko: so by no concerns I guess you're me you're fine with any of the options

Justin Florentine: oh right um I think we like option uh option three but I don't think we have a strong opinion on it

Mikhail Kalinin: okay thank you

Tim Beiko: okay anyone else

okay um next up uh this is something that's been open for a while but I just wanted to bring it up because uh we are getting close to being done with the consensus level changes uh we still have some open questions around Json RPC and how to go about like basically finalized safe and unsafe um I don't want to take too much time on the call to like go over this but it would be good to get this PR merged in the next couple weeks so that clients like can agree on what what we're implementing um

so I don't know I know like Danny Mikhail Mika like the three of you seem to feel the strongest about this uh do you want to kind of take a minute to to share your thoughts and how we should move this forward yeah I mean there's two there's two ways to kind of think about what these words mean one is the algorithm that defined that derived them or the other is like the actual status and state of the um the item I think that uh developers and people reading the API reading this would generally think the latter so like

if something says unsafe they'll literally think it's unsafe not that it was derived from an algorithm that is unsafe and thus I think that uh the problem is you can have a safe algorithm that also gives you the head and unsafe algorithm that gives you the head and thus you could have something that you could have unsafe and safe being the same uh block which I think is very confusing for end users thus I think latest is I think anchoring on the latter and actually it being like a property of that block is more is is better so I would say you leave latest I would say you define safe and you hope that you get a better algorithm over time and you do finalize um I think Justified is like a nice to

have but it would require a change to the engine API which I don't think it's valuable enough to do a break and change at this point so my argument would be yeah yeah

cool

Micah Zoltu: I agree with everything Danny said uh except for his conclusion

which is that

Micah Zoltu: um I think that when a developer asks for unsafe they're not likely to begin asking for unsafe and safe and then comparing the two they're just saying the thing I need is whatever is safe give me that or the thing I need is whatever is unsafe give me that um I don't think the user actually cares what they get back nor do I think they're going to be looking and comparing that to the other options they're just

gonna they

Micah Zoltu: just say hey I

need something that's I need something to say because I'm doing uh off like I'm taking this off chain I can run exchange or I need something that's unsafe because I am running a Mev extracting client and I need the latest absolutely I don't care if it's going to go away in group yeah but the semantics

Tim Beiko: but yeah so I think latest I think latest does represent that very well and um that people understand that when they're using latest today uh I know I know the argument that maybe it's but gone and it should be renamed but uh I don't I don't know if I want to debate this too much

today

Tim Beiko: okay I see yeah Peter and Andrew have their hands up Peter do you want to go first

Péter Szilágyi: where are these things actually used

Micah Zoltu: Json RPC API

Péter Szilágyi: yeah within that

so like get myself away this block yeah good luck yeah good luck yeah it blocked

Micah Zoltu: by number um for example ask for a block tag

there's a bunch of it

Péter Szilágyi: so I guess meaning that the execution clients should be returning these

Micah Zoltu: yeah so the as of the emerge we want you users should be able to or before the merge realistically but as of the merge at least users should be able to do get blocked by number and pass in safe as a block tag instead of So currently they can do pending latest

um earliest and I'm going under one another but anyways um we want to add finalized and save that list of things they can request and then the main question is whether you Alias and latest to unsafe and deprecate latest

Fredrik Harrysson: I mean if you video replicates latest I think a lot of applications will break um they are like a deprecation warning and not not to really get rid of it

yeah

Micah Zoltu: it's like in the docs do we say please don't use this or do we say please use this

um Andrew you have your hand up as well um yeah maybe we just call it not on sale but something like a Leading Edge or Edge a fringe her latest

Tim Beiko: um okay so I guess yeah this can kind of take up the rest of the call if we let it and I would rather not um it would be good if like people who have strong opinions can go on that PR and share them and it would be even better if like by the next awkward devs we could have some consensus on this and

have that PR merged in one way or another um because yeah it just feels like at the end of the day like in the next couple weeks is when clients are going to want to start implementing this just because we're going to want this before we we have releases out for for public test Nets um and so I think the most important thing is just for kind of being done with arguing over what the options are and I have very weak preferences for what for what the actual outcome should be but just that we should try to wrap this up in the next couple weeks

um yeah my two sense is that on that there's a

very good reason to change an existing tag to deprecate something it should be just that that is if for whatever reason it is super on appropriate appropriate than maybe but uh I know I don't really see that big applications even now latest is kind of unsafe you can have a side you can have a real one to block video if they happen so it's not like it's save currently

Tim Beiko: right yeah and I think yeah I've reached out the application and tutoring developers I think they are aware of that and like they they're not confused by by what it is

so yeah

Tim Beiko: um okay yeah moving on from this

um but yeah please comment in in in the the pr uh light client you had some comments about basically the uh the blood gas limit uh when we moved to a proposal Builder uh World um do you want to kind of give some context there right so in today's world there's these mining pools that have a lot of control over what the gasoline actually is and we have touted this in some ways as a benefit because if we needed to lower the gas limits quickly due to some situation the network we could coordinate with those

pools and make that happen and post merge the mining pools sort of go away and we have a very similar type of actor which are these block builders and they will again have this similar power of being able to set what the target limit is and move towards that I'm trying to understand if we expect these actors these Builders to be uh you know to also be good custodians of this power um and if we want to confer that power into this new system because we have the opportunity right now to add some uh

different configuration parameters to the um block building the external block building protocols that will be available post-emerge to make it so that individual validators can choose their gas Target and I wanted to ask this here because it feels like a bit of a departure from where we currently sit where we can coordinate with a fairly small group of people to change the gas limit if needed and if we allow individual validators to choose it I think that increases if I have maybe an order of magnitude and so I'm just curious about what people's take is I mean it well you wanted that argument yeah picture

that argument is just that like there should be a a lever that developers are a small set of people have and that we currently kind of do I I don't know if having more players have to be involved in this decision is necessarily a bad thing um I also think that you know validators and synonyms aren't the same as end users incentives but I do think that Builders and centers diverge even more and so putting this inside of Builders you know Med searcher's hands is is not a good equal agreement my opinion right and that's currently my opinion I would like to build it is a system where that's not

the case but I I just wanted to see if there was anybody who felt differently about that

Micah Zoltu: in the various um proposals for Builder proposer separation can we put the um block s limit in the proposer's hand instead of the Builder sends easily or is that overly complicated

uh yeah

Vitalik Buterin: I don't see why that would be complicated like there's always a a level that like there's always some signature that the proposal has to make and so you can just like add to the I guess a little bit into the container that the proposal signs so in in L1 PBS you know when there's a larger redesign

here uh certainly it's it's easier with the with sort of like stop Gap measures where you try to simulate that in an extra outside of it you know in the protocol that people are designing with Meb boost it's a little it's certainly more complicated but it's not impossible and it's not that much more complicated I'd say

Micah Zoltu: do we trust proposers more than builders

huh

Vitalik Buterin: um I would say yes because like you can like you can win and uh beneath the Builder option for a pretty long period of time just by being a willingness spend more money than other people um whereas doing that for proposers is um you know uh is much harder uh so uh

yeah like Builders like a very mature Builder uh ecosystem or with like very easily accessible options like it is more vulnerable than other Brokers choosing it right a builder could essentially buy the gas limit to increase by just out bidding all of the builders yeah exactly yeah and like unlike the censorship case where you have to win every auction if you're just trying to manipulate they got something you have to win 51 of the

oceans

Micah Zoltu: right and if you're a good Builder otherwise that may not even be that expensive because you're only paying the difference between you and it's the next best person or the best person yeah okay I think personally I'm

convinced that we should probably do something

um I'm okay with either just removing the ability for the cash limit to increase or decrease from via block Builders or

Vitalik Buterin: moving into proposers I would be happy with either personally would be a significant change that's like way too late for the merge itself at this point right so that would be a Shanghai thing well you don't need l1's for it here oh I see because like you make it like

proposers oh like clients we you're you're breaking up so you have to be forward to a validator to choose

oh sorry

Tim Beiko: we didn't hear anything

okay

Vitalik Buterin: basically the uh like you've got to have the validator um enforce it by having the validator Own Way um acceptance with a particular gas limit right exactly say at the beginning of some Epoch what their target is and the builders are just aware that that during the their proposal slot they should build towards their preferred targets yeah it's similar there's a registration mechanism to say what you want your recipient to be for a validator so essentially be piggybacking on that

Tim Beiko: yeah it seems like it would be much better if it's under

validator control than Builders and if we can do that as part of Mev boost that seems like a good way to do it now

okay I guess in that direction yeah and you want to have a strong a counter position to that so the I think

Micah Zoltu: the only the only thing question would be is um if this turns out to be incredibly difficult to integrate into because something we're not thinking about here um should we bring this back up for removing this from blockheaders with the merge or should we just accept this

builders get to control the block yes limit

I think

Vitalik Buterin: it would really be a hard very hard to make any consensus changes at this point right so the easier thing to do if we decide that we're laziest to basically add it can add a uh like one line rule that just uh says that uh proposers only accept a header if the header contains a gas element of exactly 30 million right

Tim Beiko: right then you can do that in Mev boost and not even like layer one right yeah I mean I I think yeah I think the likelihood here is it if it is complex then a version of Meb boost can be

released at the merge that does not support it and it can be layered on and if it is abused in that time frame um a software can be added to to eliminate the abuse

yeah that's fair okay

Tim Beiko: cool anything else on that

okay

Tim Beiko: um okay so uh next thing I had uh basically uh two things related to test Nets uh so first there was some discussion in Amsterdam about like the the kind of future of the test Nets

um they're a free posted a comment on the uh on on the agenda which next to it um I don't know was anyone is anyone on the call who was at that session in person I I wasn't

Marius van der Wijden: uh I was there

Tim Beiko: yeah do you want to give a quick recap or anything that's like new that came out of there

or yeah

Marius van der Wijden: uh I only attended the first half of the session afterwards I had to go to the should we break up the code of oligarchy uh session which was

way more interesting I think the big takeaway is that no one actually really wants to deprecate the test Nets and um there's some companies that want to take them over because they have stuff running on them and [Music] yeah we so the current plan as far as I understood this to bring Robson through the transition girly through the transition into polio through the transition uh deprecating could be at the point of the transition

and maybe transfer the ownership of the Rica be validators to some other entity if they want to keep this test net um and the applicator Robson after the transition probably either before Shanghai or after applying the Shanghai um Fork on it that's

that here dear goddess

Tim Beiko: and I guess just from like our perspective I assume nothing changes in that we're still happy with just running officially Robson gordian sebolia through the merge um so if if some other company wants to

like maintain Rincon B and may or may not run it through the merge um you know they obviously can but then uh yeah just in terms of like the data that we want from these test Nets these three are still sufficient and and we're happy with that right

Marius van der Wijden: uh yes exactly and so to Jenny's comment uh taking them over means that we deprecate them in our software we don't maintain them in our software anymore and they can either ah their own version of Geth or um or just use an old version where where we still have support for these

tests okay so still deprecated in a sense just kind of or from our perspective

Tim Beiko: deprecated from get perspective not from the application who has their testing infrastructure right

he uses

Marius van der Wijden: to move over to some testnets where the software that runs the test and actually gets updates and stuff like this and runs with the latest channels

Tim Beiko: that makes sense

okay

um okay and

Tim Beiko: so I guess yeah the other thing I I wanted to talk about test Nets is basically how how do we get from from where we are today to to there um so like uh we have this Shadow fork in in Dev connect uh which was which was smoother than the ones we've had before uh we have another one planned for this week I guess you know I'm curious I hear from crime teams like you know what what do we want to see in terms of like success on Shadow Forks before we're comfortable like moving to upgrading the public test Nets um and is there stuff also like outside of the Shadow Forks themselves that uh you know we still want to uh the test or to do before before we're comfortable

moving the the even the first of the public

test Nets

Marius van der Wijden: so for me seeing a bunch of successful Shadow Fox is is a really good sign um and the only other thing that I would say is uh The Hive tests should be at least for a majority of them uh passed by every client um basis currently not only have test instance but I think they're looking into fixing that um quickly get this

only failing two tests and that still isn't an open issue whether this is an issue with the assumptions of the tests or with guests um but a regardless for example is a fading 48 test and he'd be really important to get all of those tests fixed so that we have

a good confidence in the software um Iraq yeah I think we should also have Greenwood houses of course and for every client and a high test need to cover

most of the stack and what is more maybe uh fasting because right now currently in IRC framework and only in other mind and that or another is working on your passing infrastructure there's an element and death was running Netherland and peace who had

Marius van der Wijden: had some issues um but I stopped the father I think and I need to set it up again and I I also want to add uh Aragon to it and

pisu and and run the professor on all of them it's a differential fuzzle so it will also find differences between

Tim Beiko: got it anyone else have thoughts here oh Andrew yeah

Andrew Ashikhmin: um I have a question why so like the number of tests uh is different between Aragon and gas and never mind I think like the total number of tests for for Aragon is 46 for another mind it's 47 and for

gas

is 54. that's really weird [Music]

Tim Beiko: okay so definitely looking in

America

Marius van der Wijden: I'm going to look into it

Tim Beiko: yeah awesome and then uh Justin had a comment in the chat about like the mid-sync situations being like compelling and and that we'd want to like run something like that um run something like that on on a shadow Fork even though we don't have the infrastructure to do it is this

something that like client teams can manually try on the fort this uh this Thursday and see we might not be able to reproduce everything if we find bugs but we might be able to see like does everything break or do things generally work and recover um or fail in kind of predictable ways um so I don't know is is it worthwhile to try and get people to manually run those when the shadow Fork happens this week

Justin Florentine: I don't think that should be too much of a problem for

basic awesome yeah um

and

Tim Beiko: then yeah Perry has a comment about two times main net Shadow forks with no slash really minor issues um and I'm curious Barry like do you do you think our previous Shadow Fork was at that level or kind of right under because we did find these issues with the deposit processing and stuff I'd say it was right under like we're almost there but it was just yeah not perfect enough got it I'm and so obviously you know we have one scheduled

this week we have one scheduled next week um or I assume we'll have one scheduled next week I don't know for sure next week and then two weeks from now um I assume we can we can we can do those um I guess if we you know if if we had those kind of work smoothly and then uh obviously spent some time on on Hive in the next two weeks to make sure that that uh that the different teams passed the tests um are people you know do people feel like at that point would generally be in a good spot to start looking at upgrading uh test Nets and obviously you know there's always a delay by the time like we need to set the block and then like put out the software and then it's like there's a

few weeks until we actually hit the the actual upgrades but um yeah assuming I guess assuming the shadow Forks went smoothly would and and that hype support was there is is it realistic that to think uh we might start looking at testnets in like two weeks or or do we feel we need like um yeah we need much more time than that and I guess no one wants to answer this

Martin Holst Swende: um I think it sounds reasonable

Tim Beiko: thank you Martin um awesome um so yeah I I think that makes sense um

one thing I'll also share uh from discussions that Dev connect is yeah we had some chats about like the difficulty bomb and like when should we actually make a call about pushing back or not um and it it seems and this stuff is really hard to estimate so uh please don't quote me on this in two months if I'm if I'm wrong but it seems like you know we can probably get to like late May jun-ish before we really feel the impacts of it um and then um and then from that point you know in the past we've we've managed to like shift difficulty bomb upgrades in a few weeks when needed um so my my feeling from talking with like different client teams at Amsterdam was like it seems like it would be better to like wait you know try to say you know move

forward and and move to the test Nets and start start running through them uh without necessarily doing anything about the bomb and then if if we get the like late May June so like two three calls from now and we see that like we're not moving forward on the test Nets either because we found some issues or uh things are slower than expected um we can coordinate a bomb pushback pretty quickly then um but not um basically not even thinking about it and being able to move a bit quicker um in the in the short run would be better um is anyone like strongly oppose that

Thomas Jay Rush: um this is Jay Rush

Tim Beiko: hey yeah

Thomas Jay Rush: hey how you

doing

Tim Beiko: good yeah do you wanna you wanna share your screen and walk us through your charts that's what I've I've been using and I think it's yeah

Thomas Jay Rush: well I was gonna say I think that um it would be unfortunate to admit to um make a mistake there as far as um getting into a situation where you have to force yourself to delay the bomb I'm sure everybody understands that um but let me just share this one chart and I think you can see pretty clearly from the chart how quickly the bomb goes off once it goes off so is is everybody seeing this

Tim Beiko: yeah I can see it yeah

Thomas Jay Rush: so I mean we're here I I was concerned last week that it wouldn't show up because the hash rate was higher but it is starting to show up this is as of now as of about an hour ago so it is starting to show up um these are the last two bombs here and we delayed them in plenty of time for them to actually affect the block time so here's 14 second blocks um but this one was the one where we've kind of forgot to set it and we kind of had to react really quickly here because there was two hard Forks right after each other about I think they were about a month apart so

this is about a month and it just literally drops off the off the cliff so and here you can see it just drops off the cliff and this is exactly what the bomb does um so this red line is June 15th which was when we thought we were setting it back here in December and it looks to me like it's going to it's getting ready to drop off the cliff to me I'm just looking at this now I never wanted to predict the future because predicting the future of this thing is hard but I can see that we're getting to the point where it might start dropping off the cliff so I want to just say that out loud

and I want you to be careful I would rather see you guys at least pick a date to say we are going to make a decision about picking a date by this day you know say May 15th we're definitely going to have a decision about whether we're going to delay it or not because you don't want to get to the place where you're forced to delay it because it went from 18 second blocks to 22 second blocks in a two-week period that's what I would say

Tim Beiko: right and I think yeah I I think we'd probably make that call much before we hit like 18 seconds um I think yeah I yeah that sets to be pretty low but I

I guess because there is kind of this this chance that we might actually be able to ship uh the entire upgrade without moving back the bomb um probably waiting until we're in the like 14-ish second range maybe 15 to see like how we're feeling about this and then yeah and then that means you know by the time we coordinate it takes it you know maybe a period or two and we end up like maybe in the like 17-ish 18 second range by the time we push it back um yeah so yeah I that that's my I don't know rough like gauge for it I don't know if that feels reasonable sweater like yeah

Thomas Jay Rush: yeah I think I think if you set a date by which you're going to set a date you know you say we're gonna make a decision by the third week of May or something like that

yeah

Tim Beiko: yeah and I think the risk is just uh I just don't want to pay nothing to a corner we're like we're then forced to act if say the bomb is not actually showing up as like we'd expect so I if if we do you know say we can get like an extra four weeks without having to make a decision or an extra six like I I think every week where we can delay making that call is valuable to us because it gives us more information on the Readiness of the merge um yeah

so I I guess the best thing we can kind of say is like today we're far enough from the bomb being an impact that we could at least bump this conversation to another two weeks and then uh and obviously it will keep kind of looking at it but yeah yeah that would be my Approach but yeah Lucas I see you have your hands up as well

that's amazing

Łukasz Rozmej: on this graph I would say that we have um four weeks in four weeks we should decide if we should move that over or not that looks reasonable for me

Thomas Jay Rush: yeah yeah so the the third or fourth week of May or something yeah it's like

Tim Beiko: yeah that would be two calls uh yeah

uh vitalik

Vitalik Buterin: yeah I mean I think like the word the difficulty bomb is not a chaotic system right like uh you can um analyze it mathematically there are scripts that have uh predicted and pretty uh pretty accurately and like we do know how it progresses right that's like every hundred thousand blocks of the difference uh the discrepancy between the actual block time and the ideal block time doubles um and that's basically how it just uh how it keeps going right so it's uh like it is it is something that we that we can uh model the main unknown variable is basically like um a hash rate but even hash rate like it's not I mean like Unknown by yeah

some crazy factor of like more than two or whatever

uh so it's pretty

Thomas Jay Rush: it's predictable but it's much easier just to say um I I would argue that I would argue that here it wasn't very well predicted here but um and this year where it recovers so it goes off and then the hash rate recovers that that is uh dependent on the hash rate but I I agree with you it's predictable I always just predicted the first time you know when it really starts going off I'm saying this is right we're gonna start seeing it go off by the time we get here

Vitalik Buterin: yeah I mean basically like when it

breaks uh 50 and then it's uh you know pretty clear that it's a hundred thousand blocks away from breaking 17 and another 100 000 away from breaking 21 and so forth that's right yeah that's right that's right yeah um so like there's uh like it is a kind of multi-variable problem because like you know we also we have to evaluate you know the pain of uh doing an extra delay hard Fork versus the pain of uh like living with 21 or 21 or 25 second blocks for a while which is uh you know something that we have done um and uh you know in the world didn't end uh so

Thomas Jay Rush: right right yeah right and I I even I even said to Danny Ryan

Vitalik Buterin: that

Thomas Jay Rush: you should let it get long just to let the world

understand that these things can get

you know who who pushed the change to the code you know you

Vitalik Buterin: want to make it right you know ultimately like this is uh the last time that the block time is going to be anything other than uh 12 seconds uh so uh yeah I'm just I I'm you know highlighting that kind of these uh these trade-offs exist there's and there's like scripts that can help us uh actually like measure exactly what the uh what the trade-off is but yeah I mean I definitely agree with you and like no need to make it a decision now

is there a position right um Mika hey Mike I think you're next

Micah Zoltu: so we could instead of trying to project forward why don't we just pick a block time that we consider unreasonable and then as mentioned we can calculate backwards when we would need to have a you know if we know that we it's going to take us three weeks to get a release out and we know we we definitely don't want 30 seconds and we can calculate at any given point in time what is three weeks before 30 seconds I think it's hard instead of trying to calculate forward

Tim Beiko: I think that's hard because what block time we're willing to tolerate probably increases as we get closer to the merge

so we might you know like

Micah Zoltu: it's not a very a fixed variable

Tim Beiko: right like I I at least for me maybe like others disagree but for me it's like you

know if we have 20 second block times today I would consider that intolerable if we have 30 second block times but then we can shift the merge two weeks later then like that's maybe worth it um you know I'm not convinced but it's like yeah once we can tolerate the thing goes up as we get closer to the merge

um I see okay yeah that's fair

Tim Beiko: uh Lucas

Łukasz Rozmej: so two things one thing we can have this um unpredictability enhanced projectability based on hash rate leaving so I expect that close to the marriage hash rate would be going down because people will just sell their Hardware

before anyone else like before before hours um and the second thing I would really like the dream to be a reliable be considered the reliable Network and something that goes from 14 seconds to like 20 25 seconds blocks or whatever uh so I would really uh like the decision to be we can make it uh as late as possible but not before not after uh we are seeing uh time increases increases but before that so I would prefer doing that and third thing um moving difficulty pump is like easiest hard work ever so I don't think

it will um take a lot of effort from uh pushing the merge to move difficulty bomb it's very easy thing to do so those three points for me

Tim Beiko: and the first one yeah um how how quickly does the difficulty algorithm readjust the following hash rate basically

Vitalik Buterin: um I think like basically like close to immediately right because of the yeah like it can adjust by I think it's like a factor of e in um one thousand and uh 24 blocks these are 10 24 2048 or something like that uh so uh it does adjust very quickly

yeah it's not like the Bitcoin adjustment times

Tim Beiko: right so that means even if the hash rate goes down um it doesn't it doesn't make the bomb worse because then overall difficulty for the network goes down

like

Vitalik Buterin: well no it's like if the hash rate goes down by a factor of two then we basically lose one whole period until we get the

same block time

Tim Beiko: right right yeah yeah but that's by a factor of two in like a two week period

basically

Vitalik Buterin: right yeah yeah I'm uh I mean if people want to open betting pools on how much hash rate

will drop before the merge I mean I'm definitely betting that it will drop by less than a factor of two probably significantly less

Thomas Jay Rush: this is why I never predicted the future I only predict when it first comes

Vitalik Buterin: totally

Thomas Jay Rush: and I think I think I think that the the world will um tell us exactly what it thinks about 20 second blocks very loudly

foreign

Micah Zoltu: conversation because I don't care much um is the anxiety and effort we're putting into having these discussions just that alone is it worth just pushing back the bomb just so we can

stop worrying about it like like this isn't we've spent a very non-term amount of time and there's a lot of anxiety around it um I think I would say maybe value it just at getting rid of the anxiety and just even if we do still make it you know in by June just not having that anxiety along the way may be valuable in itself

Thomas Jay Rush: I would vote Yes

Martin Holst Swende: I haven't probably would too

Andrew Ashikhmin: um yeah I think it makes sense we can postpone it to whatever like the end of the year doesn't mean that the merge is postponed to the end of the year

Martin Holst Swende: no the no it doesn't sound good but the thing is with yeah as as Mark said it's trivial for us to make these hard work such an artwork that's all the bomb but it's not trivial to people to to I mean the whole coordinating the update but it might be easier if we just early on and do a plan for for bomb update and not let it affect the the merge at all or at as little as possible

Tim Beiko: so the idea is if you can if you can

push it back um uh yeah if you can push it back then obviously it's it's not like a ton of work uh there's some coordination work but um it does kind of make it smoother um Danny you have a comment about proof of work yeah I mean we might see some improved work works here but the if the chain is actively degrading at the time they're performing the proof of work work they have to do two things they have to do an upgrade and convince exchanges to list them at the same time whereas it's very easy to do the proof of work work and if you have months to

potentially diffuse the bomb and that's months to do potentially damage to users obviously I'm very biased in what I consider damage here but I I think it's actually a good thing for the bomb to be moving kind of inward as the merge

happens

Martin Holst Swende: yeah I can see I can see your point sure yeah so I definitely don't think we should like postpone it for a very long time I don't really have a strong opinions either way

Tim Beiko: and yeah I I mean we yeah we have basically a few minutes to the end of the call um does it like make sense based on just the conversations we had earlier that you know spend the next two weeks obviously focused on the shadow Forks on hive see how we're feeling two weeks from now we can see also how the bombs progressed and we can you know I think we we do have like much more than two weeks to make this decision um and like the the chain is not being affected today and it's not going to be noticeably affected in in like two weeks uh realistically we probably have maybe like even like four to make that call so like yeah I I think yeah it's it's worth at least like

kind of moving forward on on the testing seeing how far we feel we are in the process two weeks from now and and looking at picking it then um and yeah sounds good yeah if we choose to remove it or or postpone it um yeah we could we can do that in in two weeks um and obviously continue this conversation about like the pros and cons uh just from the like proof of work side uh in in the Discord uh we have two minutes anything else people wanted to bring up

okay well well finished 30 for the first time in a long time um thanks everyone for for coming on and uh yeah talk to you all in two weeks

I was promised a long meeting thanks yeah [Music] thank you

[Music] [Music] foreign [Music] [Music] foreign

[Music] [Music] [Applause] [Music] foreign [Music]

foreign [Music] [Music] [Music] thank you [Music]

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.