All Core Devs - Execution #118

Transcript

[Music] [Music] so

[Music] [Music] so [Music] [Music]

[Music] [Music] [Music]

[Music] [Music] [Music] [Music]

[Music] [Music] [Music]

Tim Beiko: and we are live uh welcome to

awkwardev's number one one eight um i posted the agenda in the chat already um lots to cover today i guess the first thing is obviously the robston issue that happened i think on wednesday um i have a quick summary of it that i've uh i've shared a few times before but at a high level uh there was a transaction which was sent from an account i think which would did not have enough funds to cover uh the max fee for its transaction um and because of

how the assertions were done differently in geth aragon and nethermine versus basu and open ethereum that transaction which should have been invalid got mined into a block produced by geth and then it was accepted by aragon and nethermine rejected by bayesian open ethereum um and because geth was the only mining client on robson at the time and they were basically the only ones making the chain progressing and so everybody else kind of got stuck on the the previous block um yeah so and uh yeah it's you know it was pretty quick to identify the fix

um i think within one or two hours of of somebody noting the issue um you know there was a pr open and geth with the fix and then pretty quickly after in eragon and nethermind and then yesterday those three teams put out a release with a fix for this um the one thing that did take a while that i noted is it took about six hours for somebody to notice that robson was stopped but i guess this was to notice that you would have had to be running basically an open ethereum node um or looking at ether scan which seems to also be relying on open ethereum um yeah so i don't know i guess just to

start off if anyone has any comments generally or context they want to add around this if not um i guess uh lxi had some comments um in the awkward agenda uh so if people want to scroll down i won't read the whole thing um but basically one thing he mentioned is uh the fact that to resolve this uh to repair the robson nodes uh miners have to run a full sync rather than like a fast sync and we should try to think about

you know what would we do if something like this happened on mainnet um especially in a spot where say geth is the majority client they're producing blocks it obviously takes several hours to have a fix out yeah and i see martin has his hand up

Martin Holst Swende: yes so this is not unprecedented we've been here before and when that happened the last time uh don't recall exactly which time but we had these long long chains and i think it was also robson and it was a pain in the ass to sink because when guest and i guess all the

clients do the past thing in snap sync we just select with the heaviest one and we just jump ahead to a state close to the head and thereby yeah we wind up on the wrong side of the fork because the the that's the highest one and what we did back then and guess is we added this uh command line flag called the whitelist which is a comma separated block number to hash mappings to enforce so you can basically start guess saying add block number something i want to speak i want to have the hash something and when guess then peers with someone

else geth will ask what what hash do you have for this number or rather give me this number and it checks the hash if it is what it expects and that means that you get will basically enter a mode where it only will speak [Music] on the same on the desired chain uh yeah it's a clumsy i mean it's it's not ideal it's pretty bad ux but it's otherwise kind of hard to know i mean the operator needs to step in and say i don't actually want the highest difficulty chain i want

the other one and yeah i don't really know how we can make the ux better but this is this is kind of the best idea we had at the time so it is possible to handle this scenario um yeah if anyone has better ideas i'm hoping to hear them and if not i think it would be good if other clients also have something like this

Tim Beiko: so he's not here but alexa i said aragon is working on a way to go back before any black block in the past

and kind of i guess

Martin Holst Swende: yeah sorry so we don't set head in death and one thing we did uh consider previously was if if we should at like the fourth block minus one um if if there's like a known fork should we do an explicit flush of the state so you can always go back uh that can be done but it's kind of might be a bad idea if the entire if all the nodes on the entire network does an operation at the same time even if it's a network because it can have macro effects where

the entire network uh becomes slow although i don't really know how otherwise to do it if you have an un uh intentional fork uh i mean i either you you always flush and you basically run in archive mode and you have every state always uh other than that it's yeah how i don't know how how one would implement what he says other than what we already do i mean you can do a set head and yes uh it will right rewind to whatever state you have it might have to go back 4 000 blocks to the last time we flushed it might have to go back up to like two

hours from processing time uh but that's kind of yeah that's where we're at and i don't really see how we can improve it

Tim Beiko: got it um marius you also have your hand up

Marius van der Wijden: yeah i just wanted to say that it's not not 100 percent accurate what what alexa said um because i um did uh set had the uh before the fog block with my node that was snap sync so it's it's definitely possible if you have i think when i when i when i heard about the fork

i set my i immediately set my node into archive mode to sync the chain and so i had the states available before uh set so it's basically it's not it's not really part of the the synchronization mode but if you have the state available and because i had the state available i could uh actually uh set that before that

Martin Holst Swende: but yeah and it's basically you can wind up in two things can happen either the fork either you were thinking or i mean the fork has happened and now you want to sink

you want to avoid landing on the wrong chain because your pivot block will be after the fork and you can use the whitelist block for that the other case is that you did sink you let follow the wrong chain which means you did sing so your pivot block is earlier than the four clock and then you fold the wrong chain um you discover oh i'm on the wrong chain you update your guess and now you can always do a set head back to the pivot block you're i mean you're guaranteed to have the state for your pivot block and probably some other nice locations afterwards but so both of those situations

are resolvable with

guests

Tim Beiko: would it be possible for either of you uh marius or martin to basically add a comment by that in the in the postmortem so i'm gonna i'm gonna move the postmortem from my hackmd over to the eth1 specs repo uh today um but i think it would be valuable if we can just have like just marius maybe explaining kind of what you did on your node um because if in the future something else like this happens then at least we'll have like uh some instructions about how to deal with it

Martin Holst Swende: yep cool i can write on the agenda

okay yeah that would be great then i can

move it over to the postmortem and make sure it's documented there

Tim Beiko: um did anyone else have just like thoughts comments about the robston issue oh i see we have jeffrey oh no

there

i just wanted to uh chime in um i'm from the group that actually mined the block that had the problem and uh we were working with trying to figure out how to come back and fix it and um i would just say that the instructions there was just a lot of we weren't sure should we do a debug set head why wasn't it working so i think just kind of like a flow

chart that would say what to do in this situation for death miners would be uh extremely helpful but given with what geth already had in it we were able to repair it and get the mining uh and get the chain going again

Martin Holst Swende: yeah one thing i think is might be hard is knowing what is my pivot block and how far can i go back um and that yeah i i i yeah it's it's not trivial yeah and

currently the experience is guest sort of just either crashes or you get error messages that don't tell you what so uh just maybe just a in case of

emergency break this glass sort of post in the postmortem uh to tell miners what to do i think would be sufficient

Tim Beiko: yes yeah thanks for sharing jeffrey anyone else have anything they wanted to add or bring up

Marius van der Wijden: um i think the fork is not really not not really resolved yet so if you have any spare mining power to to work on the real chain then or if you still have mining power on the old chain if you ran a miner before the fork happened and you didn't hear about

the fork yet then please shut down your miner update your node and mine the new chain with uh with us

yeah and it's worth knowing so for robston i believe geth and basu are the two clients and open ethereum are the three clients actually that can mine on it any other comments

Marius van der Wijden: um yeah we should also update the um the sealers on all the other chains

um two versions that have to fix

Tim Beiko: yes that's right uh so for ring could be ice i think the get team has all the cedars there is that correct

um no we don't have all the see there's something to be

Tim Beiko: okay so i guess to any seeders on rinkyv you need to update 1.10.6 correct yes okay and then for gordy um there is like a wide set of validators across a wide set of clients um so just

for this to be clear uh if you are a validator on gordy uh using geth the version is 1.10.6 if you're using aragon i'm not sure there's any of them but still at 2021.07.04 alpha and if you're using nethermind um v1.10.79 um all of these are linked in the postmortem obviously and i'll make sure to have an update in the london announcement blog post on blog.ethereum.org later today which has updated versions listed there any other comments

okay so i guess one thing that's been kind of implicit but i just want to make sure there's no objections or thoughts about is um the fork block for london so all three releases from geth nethermine and and uh aragon uh kept the the current uh block uh for uh august fourth or fifth that we're looking at right now um does everyone just still think this is the right approach uh obviously you know we found this on a test net which is good um and we've added tests to to catch this um yeah do people still feel comfortable or does anyone have a concern with going forward with the current block

once twice okay um okay so then for anyone listening just to make it clear um you know the fork block for magnet for london hasn't changed um the versions for geth aragon and nethermine have um so you can see them in the agenda for today and uh i'll make sure to update the london announcement blog post and share that on on social media and whatnot uh with a note about this uh just so it's clear cool um so second thing on the agenda um this is is my command yes um so

on the last call we discussed uh basically the gas price field uh for 1559 transactions and um to recap the implementation is 1559 transactions contain a gas price field right now uh in order to maintain backward compatibility with tooling um but because you can't actually know what the gas price is of the 1559 transaction before it mines um pre be in mind the transaction's gas price will uh will reflect the max fee per gas and after it's been mined it will reflect the effective gas price which is the amount that it actually paid based on the the current block space fee um mica is if i

understand it correctly is it's very bad to have an api which can return a field basically which can return a different value based on time and the state and that'll lead to it being very hard to debug um micah did i get that roughly right

Micah Zoltu: uh yeah i can finish up if you want sure so yeah so the what tim said is basically the the short summary um and any time you're debugging an issue and the behavior changes based on when you look at it it becomes a very very hard bug to debug i suspect that most users and app developers and library authors and whatnot

probably are not watching closely on these things and they will not realize that there's a change in behavior in the gas price field and so this will be an implicit behavior change which if we accept should at least be something that doesn't result in like this really long tail of hard to debug problems where like my app kind of works sometimes doesn't work other times every time a user reports it then when i go to check later it works fine i can never reproduce myself like that's what we're going to see if we go forward with this kind of behavior change i think my proposal is to just have as an alternative is to just have the max fee be always returned as the gas price even after transactions mined because at least that way like you have

consistent data being returned and if adapt breaks or something because of that they can easily figure it out and fix it unlike the current behavior which is the dap will still break just it will break sometimes and only for some users and only in some contexts um so yeah so and then that and i would also like to i believe you guys talked about deprecating this field it'd be nice if we put an end-of-life date on it like maybe the first patch after shanghai or something i'm just so people have like a clear timeline and they know you know is this something that i need to deal with removing from my software soon or do i need can i put it off i think just saying is deprecated for a lot of people means oh this is going to live

forever because sadly in software that's very common um so those two things uh i'd like to get the actual end-of-life policy whatever that is don't care but i think there should be one and i think we shouldn't have this behavior where the return value changes based on time and context

Martin Holst Swende: i disagree

Tim Beiko: on which points on both or just on the value changing

Martin Holst Swende: um well we do have an epm op code called gas price right unless i'm mistaken okay sorry say that again you have what

an evm up code for gas price uh yes but we don't don't know what that's going to return until after the transaction mines yes but during when the transaction actually is included in the block we do know um and i i think it's i don't see why we would not include that information for a transaction where we do know the effective gas price

and use it as gas prices

Micah Zoltu: so the long term i think the right place for that sort of information is in the receipt like so get transaction receipt returns values that are only

available after um mining so includes the effective gas price and um status for example and there's a couple other things that are not known until afterwards and that gives that makes it so there's very clear behavior for app developers and so they know that hey if i want some piece of information that is only available after mining then i get that from the receipt if i want information about the transaction itself which is available before mining so basically the stuff in the transaction and i get that from like getting transaction or whatever i think long term i think that's the right behavior do we agree on on that at least

Martin Holst Swende: yes

okay

Micah Zoltu: so the question then is left what do we do in the meantime so we

don't break people um i'm generally convinced that we shouldn't just return nothing for that field because there are a lot of people that use gas price for things and you know if we just didn't return it for one five five nine transactions those things tools would break i feel like though given the two options of return two different values depending on whether the transaction's mined or not i think that one's going to result in a lot more pain for app developers just because the debugging problem i guess the question is do we want to do we want a solution that is easy to debug

but more likely to fail or do we want a solution that um fails less often but when it does fail it's much harder to debug i think maybe that's the trade-off here

Martin Holst Swende: yeah and i think maybe uh us client developers are not the ones who have much at stake here i mean it as a guest developer it's like we don't really care we do this way or the other way but this might be important for people like in fiora or people who are actually using apis a

lot

Tim Beiko: so yeah i guess two thoughts on that one um next friday trent and i we're going to organize another infrastructure call

to get basically these projects who are adding support to 1559 and london in general right now and and you know answer their questions um so that's definitely something we can bring up there um the second thing is basically every client uh has a version out for london now that has this kind of changing behavior implemented right where it goes from um it goes from the max fee to the effective gas price um so i suspect there are already some projects that have started doing this and it might be the worst of both worlds if in some minor release you know geth or basu or whatnot

changes the behavior um because then it's like you'll have you'll have some versions that have like you'll have basically a different behavior based on which non-consensus breaking change version of a client you're using um so i suspect if we really do want to make this change um it's almost like you need to i i don't it's like you need to tell people use this version for the hard fork in a way because otherwise you'll be in a spot where there's just a lot of inconsistency

Martin Holst Swende: but yeah but related to this aren't there also other fields which are

added for transactions which have been included in the

book

Micah Zoltu: uh yes like the transaction index and yes number number yeah

uh i mean all those three i would love

Micah Zoltu: to see all those i would love to see all this move to the receipt for the reasons i mentioned before like having the transaction return like a static set of fields that are define the transaction and then the receipt return all the things that you know later and that would be my preference long term um i don't i'm okay with like a dragged out deprecation strategy for moving to that um because i recognize it's not easy to

just switch yeah i mean yeah

it's easy to switch but then whenever we do api changes i guess we a lot of people get angry and yeah that's the problem

Micah Zoltu: so if we do go forward with the so switching contacts a little bit okay is everybody okay with setting an end of life a clear end-of-life policy like a tentative date such as first um release after shanghai will remove this thing or whatever that is that would make me at least a little more comfortable because it makes it

more clear to people that hey the real solution here is to stop using this field and it is going away in a real time like i said just when you say a thing is deprecated i've found that people usually continue to use it until there's an end of life policy in place because in software people deprecate things all the time and they stick around for seven years

it

Martin Holst Swende: sounds good to me

Tim Beiko: and micah you're proposing basically waiting until the release after shanghai so that means that uh the gas price field would be supported basically up to the the upgrade after shanghai right

Micah Zoltu: um without having to lose access to that field however if you want to run the client that's compatible with the next hard fork then you're gonna have to fix your stuff

Tim Beiko: yes exactly but we're not asking people to fix their stuff before shanghai basically we're asking them to fix it correct right after shanghai

Micah Zoltu: yeah so sometime between shanghai and the next hard fork if we if we chose you know patch after shanghai then that means that people need to fix their stuff sometime between shanghai and the next hard fork so that gives them like a six to twelve month window depending on hard fork timelines there's a strong opinion on you know

time i hear more just that there is one

Tim Beiko: what i like about shanghai is basically we need to have a fork around december this year right um so i like that we can kind of put a date on it already because of the difficulty bomb so i you know and it's also like far enough in the future that like clients their applications don't need to deal with this today but it's you know not that far that this sticks around forever so i i would be fine for keeping this until basically the difficulty bomb pushed back and removing it in the release after

Micah Zoltu: okay so how about then uh for a path forward so we can move on to the meeting

um we will tentatively say patch after shanghai is the end of life for this gas price field on one five five nine transactions it will still remain on legacy transactions because it actually is part of the transaction there people can discuss this in all core devs channel and discord if you have additional thoughts after the call separately let's talk to the app developers and the call you mentioned next week and see if anyone have strong opinions if we're gonna be deprecating this in the next like six months i am more okay with just having this weird kind of uh behavior that we currently have just because i know it's going away and we can just tell people when they're under these bugs sorry anyway we did our best to give you something that kind of

was a stop gap but you really shouldn't use this field

Tim Beiko: yeah i think that makes sense um and i'll also bring up the timeline on the call next friday with them to see if anybody has like a strong reason why that would be too quick or something cool anything else on that for me okay um so next topic oh right sorry go ahead

Rai Sur: yeah actually before we wanted to move on i just wanted to ask a question about a possible bug and geth uh

after main networks i i was just curious what people's expectations are for the dynamics around miners and the community if there is a bug in gas as someone who's relatively new to his face wasn't around for what i heard was a bug getting enshrined in parody i just wanted to understand yeah what how people think this would go down

Micah Zoltu: poorly

Artem Vorotnikov: um i think we should just recognize the current reality and make it so that if there is a consensus issue and death is in the wrong then everyone else corrects the way so that it is the same way as geth and we just

edit the yellow paper to account for the quirk for specifically for gaff

Micah Zoltu: it would be great if like i kind of agree with artem that i suspect that is how it will go in reality um it would be great if we could avoid that though um just because it just means we now have this technical debt that we're gonna struggle to get rid of for probably a long time depending on the bug some bugs you know you can pay off a technical deck no next hard fork or whatever but i do worry about introducing technical debt because of a bug introduced

in death that would suck

Artem Vorotnikov: i'm afraid it will be too painful for the mainnet especially since maine and by the time we

recognize the bug finderbug the mainnet will basically chew billions of dollars but by the time and uh it will be inevitable that this book will just be written into history written to the annals and recognized as a quirk in ethereum history and that's it

so

Martin Holst Swende: um there is another i mean yeah there is also an added complexity i mean even if we were to say that yeah artem's proposal that's what we're gonna do uh they can also be that the the bug is only in geth

latest version and 40 from that or 50 or 60 but in older version the bug is not there so it's not only the case that we may have you know guest splits off from the rest we can have guests piss off from geth and the rest so we still might get very complex scenarios

Artem Vorotnikov: and i think it should generally be go with the majority chain and that's it

Martin Holst Swende: yeah i i would not commit to do that in right before knowing what the bug is

because the if we decide to leave a bug there it might have it might be able to you know to be used in ways that make it a denial service vector it might be able to without knowing what bug is it's kind of hard to

that

Danny Ryan: that the majority chain could have printed 100 million ether and we probably will not follow that

Martin Holst Swende: yeah i mean we we do have a k we know that there is there was a bug in both gap and party which was discovered after the fact and you know the ripe md touch delete which

was not deleted because it was a precompiled uh and to say that we'll just amend the yellow paper uh that we haven't actually managed to do that we haven't fully been able to describe the scenarios where where you know the this behavior um how to describe this behavior is it something special about ripen d is it something special about the whether the rollback happens in the outer transaction or in an internal transaction or it's something magic with pre-compiles it's it's it can be very difficult to yeah retroactively

so i don't know i i think it's difficult to i mean it's good that we plan it and discuss it but i don't i don't really think we i cannot really commit to any particular strategy right beforehand before knowing more facts about it

Artem Vorotnikov: this discussion it would be so much easier if we had a thing like regenesis on the table where we could you know uh not uh not fix go with him not troll back the main net but instead uh have this quirk enshrined and then have regenesis after some time which would erase that quirk from the code of all

implementations where we could fix uh fixed implementations remove the past complexity by just setting a new genesis block basically with the current state

Lightclient: i think that's what we're looking into

the state x3

Artem Vorotnikov: there are two parts uh to this proposal one is just regenesis basically sitting the new genesis block and the other one is expiring is shadowing the inactive state the state expiry these are two parts of one big proposal basically

so

Micah Zoltu: on this vein of topic

do we have any ideas or thoughts or plans or strategies for maybe trying to get more client diversity in minors or validators in the future or maybe danny knows is do we expect more client diversity in validators for consensus

proof of stake

Danny Ryan: um i think we have a bit more diversity right now um and it is still trending in a positive direction but uh it's hard to say

Gary Schulte: it's probably to be determined yet what the execution client distribution is going to be like that's right only paired with uh

you know the consensus client

Tim Beiko: yeah and yeah i guess ideally we would want to even i was just going to say i suspect because of like switching costs right there's not like a magical thing that will happen where the client diversity resets after the merge just because um if you're already using geth or you're already using nethermind um and the goal of the merge is to minimize the amount of uh of of i guess changes for application developers then you know um yeah you'll be in a spot where you you don't really have to change

yours

Micah Zoltu: yes but i think we actually do have a

little better client diversity in non-mining clients um partly because not all clients can mine and so like another mine for example there are zero mining not my clients

Gary Schulte: but there are non-zero um non-mining the actual players in uh in the merge are going to be somewhat different it's not going to be a domination of just a handful of pools

Tim Beiko: yeah and uh william had a comment in the chat about mev right like that's another uh big factor in the client diversity so uh clients that are obviously optimized to capture mev um there's just like a higher incentive to use them and especially so

after the merge yeah so i first i i guess any other thoughts comments on on rye's original question

Rai Sur: do we think there's like is there some number n number of hours that we think miners would roll back even if we didn't like if we if we did had have uh the worst case where geth doesn't disagree with older death clients and everyone goes one side or is it kind of like as soon as it happens no matter how fast the fix goes out

uh

we're screwed

Micah Zoltu: i think as martin indicated earlier it depends on the severity of the bug like if it was a bug that printed 10 million eth then i think we'd probably roll back no matter how long it took um if it was something like you know it's it's annoying it caused a desync but things continued functioning properly then it's more likely that we'll just roll forward with it

Tim Beiko: yeah and it's also i guess one kind of factor that's worth noting is just like the time that it takes people to move

eth right like the amount of deposit confirmations on an exchange for example or you know the time that it takes to make like a uni-swap trade is on the order of like minutes and even for the most severe bug you know you're on the order of hours to find the bug and put a release out um so there are you know in those hours like artem said there's like billions of dollars in settlements that happen um so if anything kind of changes the history there uh you're in a spot where you've changed stuff that you you basically can't undo right like you you're basically double spending on exchanges and um undoing a lot of uh and doing a lot

of other activity that's happened on

chain

Martin Holst Swende: yeah but i mean there's always been been the notion right if you if you run an exchange you should run multiple clients and you should detect when they are not in sync and halt the deposits or withdrawals and take steps yeah we've always been open with ethereum that there can be forks and there will be consensus issues and there's the possibility of that we don't have the reference client

Artem Vorotnikov: to be honest i think that even in the

case of you know someone printing a hundred a hundred million if the better way forward would be to just have something like that dow hartford number two than to roll back the chain because if we roll back by even by hours this would completely destroy ethereum ethereum's reputation basically as a reliable platform

dao hard fork

Danny Ryan: number two wouldn't necessarily be possible that i think that was uh possible because of the peculiarities of the the capital being locked for some amount of time whereas many versions of such a bug would you know result in many transfers uh center exchanges you know swap trades

all that kind of stuff so i really i mean i don't think that we can come to a like conclusion as to how we would proceed i think we there are just many options for the many types of bugs and many types of peculiarities that will emerge

Artem Vorotnikov: yeah just to briefly point out i mean there's nothing that's very like that's particularly special about along this chain right so as soon as the kind of the rules the chains follow diverge anyway then some exchanges might be on one chain and some extent it might be on a different chain and um it yeah to me to me it seems like it was always kind of the obvious part of the social contract around the theorem that

if you end up on a chain that does not follow the ethereum rules you're just not on the canonical ethereum chain and like maybe potentially if it's a super small um difference to the cannot to the conical rules and it doesn't matter it could be that we end up just kind of accepting that as the clinical chain retractively but i don't think it makes sense at all to kind of have any standardized expectations wrong around that behavior

Martin Holst Swende: i have one more comment yeah uh just um is it so easy instead of just generalizing uh so let's say for example this part that

have robson if that had happened on london um i would totally accept if we just decided to change the rules and allow it because the only side effect would well a transaction made it into a block uh the transaction did fully cover all its own expenses it had the balance for the effective gas it just failed a check which is not really that important and i would totally be in favor to just okay we don't have to actually rewind maynet

for that but yeah i wouldn't make it a general

rule

Tim Beiko: yeah that's a very good point ryan do you feel like we've answered your question or does anyone else have any comments or concerns about this

Ansgar Dietrichs: yeah maybe i just thought i wrote it up i wanted oh yeah go on oh sorry i just wanted to have a good question is would there be room for like

some sort of oracle system to quickly kind of inform stakeholders that there's some sort of consensus issue like i mean of course ideally exchanges and everyone should run multiple different clients but um could there they i i don't know that it makes any sense at all but could there be some way to kind of like very efficiently inform people but now it would be time to be very careful

Martin Holst Swende: yeah we could i mean something like pork mon if you could subscribe to events from the fork one for example or if people built alternatives to the fork one um that's what is there for you

it

Micah Zoltu: sounds like a great

product idea and business opportunity for someone listening on the call

Tim Beiko: yes and i suspect uh a lot of exchanges or other infrastructures basically have something like forkmon and that says if you see a bad block just pause everything until a human investigates

Danny Ryan: um i would say some rather than a lot based off of my conversations

Martin Holst Swende: i'm fairly certain that the inferior has one um and if anyone wants to add the feature to the existing port one um the repository i think has been moved to ethereum so it's like node monitor um prs are welcome otherwise we'll

we can try to extend it ourselves it might not be yeah it might not be done next week but surely something we can look into

Tim Beiko: know there's a core dev apprenticeship program going on right now so if you are a core dev apprentice uh this seems like a pretty low hanging fruit that provides a ton of value to the ecosystem and if you're not sure how to get started just ask in the discord and somebody will help you cool any other thoughts comments okay uh yeah thanks a lot for bringing

this up right i think this was a pretty valuable conversation to have um okay so next up on the agenda another big topic um so we talked a couple calls ago about potentially working on shanghai and the merge in parallel um on the last call we briefly touched on it but not not too much um and i guess i was just looking at dates and it feels like at a high level we basically need to decide either today or on the next call if we plan to do more in shanghai than just a difficulty bomb pushback um

the reason behind that is obviously if we just do a difficulty bomb pushback this is not something we need to deploy on test nets before mainnet because they don't have a difficulty bomb it's obviously kind of a one-line change it's something that's easy to test and whatnot so we don't need to like add complex new features in clients um so you know we basically don't have to decide if we do it and for how long probably until october or something um on the other hand um if we did want to include new eips alongside this difficulty bomb pushback which has to happen sometime early december um then we basically need to have the final list of eips chosen

like in the next two weeks so that we can then start implementing them testing them having an upgrade on test net and then having an upgrade on mainnet and it's also worth noting um you know if we do want to focus on the merge and client teams can't do everything and so um it's not clear that you know it's not clear what's the capacity to say do those things in parallel and how much that would push back the progress on the merge um so i guess yeah i'm uh and oh so yeah so there's a comment in the in the chat about saying are we certain that the merge this year is no longer an option

we don't know um but regardless of that you know it's like if we work on something else and not the merge we probably push back you know the merge more um so yeah i guess i'm curious what different giant teams think about all that the idea of like shanghai versus kind of mirror glacier part two versus the merge um yeah

Artem Vorotnikov: i think we should do shanghai

Tim Beiko: and by shanghai you mean like a feature fork

Artem Vorotnikov: yes a future for it i mean there is at least one eip that i would like to see in that is the ethereum object format

for edm but maybe we should also include something else

Martin Holst Swende: um are you speaking for aragon i mean the selection also feel that way do you know

Artem Vorotnikov: right now i'm speaking for me personally but i would suspect that that would be his position too

Micah Zoltu: one thing to keep in mind with the ethereum object format is um it keeps coming up as a potential long-term solution for some hard problems like state experience and so it is part of

um you can consider it i think part of long-term uh feature development like it's an iterative step but it's not just like a one-off like little thing it's like it enables some bigger things down the road this is eip number i don't know off top sorry 3670

oh

Tim Beiko: thomas you had your hand up

Tomasz Stańczak: yeah sure i think it would be great to uh keep pushing for the test net for the merch in october after some of the workshops and meetings so i know it would be a very short term but at the same time we we expect that

end of september some big push should happen from all client teams and then um i believe that if destined happens then there'll be a period of time when many many infrastructure providers and users and apps will be experimenting with it while decoratives may refocus for a moment on the shanghai fork and i would probably be happy to see it to introduce some of the eips that were proposed and i was looking today in this object format to remember that many teams were suggesting it was important and i think it might be reasonable to take it into account

Tim Beiko: so basically just to make sure i

understand you're saying you know we get merged test nets up um as soon as possible you suspect we might be able to do that before october and it's like when october comes then we and the test nets are up then you think we would have more bandwidth to work on you know kind of a feature shanghai

well

Tomasz Stańczak: i think that's maybe end of october test that is more likely and then if we if you manage to keep shanghai reasonably uh slim and the like outside of core devs will be enough of the analysis of those aips that one or two may hit the shanghai and the other ones may be proposed for cancun for next year

got it um base you any thoughts

Rai Sur: sorry i was just tracked by something what was the question

Tim Beiko: uh any thoughts on basically shanghai versus the merge versus your glacier part

two

Rai Sur: for december

Tim Beiko: yes well i mean not necessarily committing to like having the merge in december but whether you know our focus right now should be on you know just focusing on the merge trying to

uh potentially add features for shanghai and parallel or yeah any other thoughts

Rai Sur: i'm leaning towards focusing on the merge

Danny Ryan: okay i have a couple quick comments um one i think getting the other side of london spending some time with the new merch eip and accompanying other specs um through september is going to give us probably much better information and i also will say that a number of researchers and developers

certainly are stretched then and committing to a fork at the end of the year and committing to driving head first towards merge um at the same time seems like committing to a lot of work that we often kind of get into and then realize we have too much on our plates so yeah

Tim Beiko: so it seems like i i agree we tend to over commit to stuff and even you know i think london was an example where we tried to keep things small and it kind of grew um so i suspect whatever we commit to will end up being bigger than what we we kind of agreed to up

front um it seems like everyone is is kind of aligned on focusing on the merge and there's some disagreement about you know when and how we might be able to focus on something else um so perhaps it probably makes the most sense that you know obviously get london out the door um which will happen uh which will happen in the next couple weeks uh then spend time going over kind of the merge requirements and whatnot then we can maybe make like a more informed decision after that um and it's just worth noting that yeah um at some point it becomes kind of unfeasible to implement test deploy stuff on testnet and deploy it to mainnet um but we don't necessarily have to make that call

today

Danny Ryan: and a couple of ago ago we discussed queuing up a feature fork in terms of vetting proposals and kind of figuring out what that would look like and i would say regardless of when shanghai does land it's probably good too we can have that conversation in parallel to the merge without having to do an incredible amount of technical work on it um and so the overhead effect planning for that is probably low whereas the overhead of actually releasing it on a certain timeline because that's when you uh a lot of the work comes

Tim Beiko: yeah that makes sense uh micah you have your head up

Micah Zoltu: uh just as a reminder shanghai will happen this fall because we have to move

the difficulty bomb it's just a question of whether it has interesting things in it besides that um also do client developers currently have an idea of how difficult the evm object format implementation is this is like one of those things that's like a trivia trivially easy or oh this is a big project

Martin Holst Swende: marius said i'll go ahead now so we talked about one of them called validation which is 3670 which is kind of small and neat but it also requires 3540 which is the ufp one um i think well i think it's non-trivial to

implement and i think it's non-trivial to make test cases for but i i mean i i'm i'm fully supportive of it and i think it's good and it's going in the right direction but yeah i would prefer to focus on the merge for the next [Music] that's very valuable i was looking for just like a gut feeling on complexity

Tim Beiko: and i guess yeah you had another point about like shanghai will happen um you know in december uh aksak had a comment about that where like you know maybe for clarity um if we

want to for you know if if we want to target a a difficulty bomb only fork in december maybe we don't call that shanghai and we call it uh some like two-word name like we did from your glacier his proposal i think was arctic glacier or something like that um so that might be like an easier way to differentiate between them kind of saying that you know the merge is obviously kind of brilliant proof of stake over to the current evm chain um maybe we have shanghai which is the set of features and we have you know something else called it arctic glacier which is just a difficulty bomb push back and two of those things might happen at the same time right like or i guess if we were in a spot where the merge happened in december

we just don't need this kind of mere glacier part two um but if we are in a spot where in december we feel like you know we want to do both the difficulty bomb push back and add some feature um we can just have you know kind of merge those two hard forks but have shanghai only refer to like actual functionality improvements um so that it's clearer for eip champions like you know like we maybe don't know when shanghai is but we know that you know these set of changes will come together on on on the network i see there's a plus one from martin in

the chat is that about having kind of the three different names like merge shanghai

and oh no it was common to andre but oh there yeah just an implementation of 35.20 which is cool really good

Gary Schulte: i think that's the good naming distinction

tim

Tim Beiko: okay so yeah i guess does anyone disagree with having the three names and worst case if we do have shanghai which features at the same time as this difficulty bomb hard fork we can just merge those two

specs into one

Micah Zoltu: i do but i've already expressed them many times before and so i don't need to reiterate them

Tim Beiko: so just to be clear you'd rather have to be called shanghai no matter what's in it

it's

Micah Zoltu: in the fall of 2021 we don't know what that's gonna contain but that's what we call the thing that happens in 2021. it's a very different strategy that's my strong preference but

Tim Beiko: thomas you have a ambiguous comment in the chat as well so

which option do you prefer

Tomasz Stańczak: so one question because you keep saying the fall um isn't it not to happen in december

yeah which would be winter oh sorry it depends on your region i guess yeah that's how i wanted to confirm it's just a matter of geographical approach

Tim Beiko: okay so thomas you prefer calling whatever happens in december shanghai right

Tomasz Stańczak: yeah i think it helps community to understand that they can at least vote for things to be in there or not and when it happens chronologically

Tim Beiko: okay i don't have a strong opinion so if people prefer that to keep shanghai i guess william there's one opposition to keeping shanghai referring to the future fork but

Marius van der Wijden: um yeah i i would also oppose i would also say that um it's cooler to have the three different names and and merge them if needed and to keep shanghai as the feature fork uh regardless of whether it happens in in december or q1 2022

Martin Holst Swende: um yeah i would oppose the opposition i i

also lean towards having whatever whatever is in the next fork and we don't know what's in it but it's shanghai it makes it i think it makes it easier

Tim Beiko: okay i guess it seems pretty split um do this like a weird way to phrase the questions but do people kind of is it worth bringing this to say like the infrastructure caller and awkward devs and actually talking more with people who propose eips than with core devs and kind of see what they prefer um because i guess it feels like in a way you know we will kind of figure it out

no matter what it is and it's kind of our job to stay on top of this thing um but maybe kind of the perspective of the people who are actually submitting stuff into these is you know what makes it easier for them so this

Micah Zoltu: yeah i think it also has an impact on just the broader community um people's people often talk about london for example long before london comes out and um so i think i think they also are stakeholders that matter like it shows up in news articles it shows up in chat rooms it shows up and twitter tweets like these names show up everywhere and they affect a lot of people so

you

Tim Beiko: so with that being said you think the time

like in shanghai in the in december is it's just easier for people to keep track of

Micah Zoltu: that's my feeling but like if we wanted to solicit external um opinions i think that we should extend beyond just eip authors and also try to see if we can get feelings from the broader community which i know is extremely hard

sorry but

Martin Holst Swende: didn't we didn't we kind of decide to use the the devcon sites as the name for hardworks just so we wouldn't have to be by shutting about names so much in the future i don't really see why we should deviate from that on that

like when we did

Pooja Ranjan: istanbul that was the first time when we decided

that we will be following that defcon names and right after istanbul we did new glacier because that was just the one single fork to push the difficulty bomb and i feel that it makes sense if it is only for difficulty bomb we should have something like uh that was proposed like our arctic glacier just to be in think that okay fine it is just for difficulty bomb and for all other feature folks that are very important for people to upgrade their notes especially like oh with the little version client it makes sense to go and with it upcoming i mean that's just my thought and it's not like very very

hard and yeah it's not a hill i want to die on i

actually i think i can accept whatever action yes

Marius van der Wijden: for me it's also not a hell i want to die and just um my idea would be if like we start pushing for a feature fork called shanghai and we then realize that it's or like and we say shanghai is going to happen in december and we then realized that we cannot really do the future in december and then have to i don't know either kick the eips out of the out of shanghai i think then naming the the feature for shanghai and

moving it would be better in my opinion than [Music] basically uh taking out all of the eaps that we said are going to be in shanghai

Tim Beiko: yeah i i think the the way around that is we just don't commit anything to shanghai you know uh uh until we get there um it does feels like yeah it's starting to get like like um i'm not sure there's yeah i guess yeah i'll i'll start a thread on each magicians um just to get kind of broader feedback on it um and if

people have strong opinions i think this is kind of like the last time we could voice them um yeah i don't think it makes a lot of sense to discuss much more on the call like client do you have your hand up oh we can't hear you if you're speaking oh sorry about that um if you want to come back with your comment when your mic works uh we can we can have it later um okay so uh to change gears completely from bike

shedding uh this week uh mikael opened a proper eip for the merge uh so eip3675 um and you wanted to take a couple minutes to just kind of present it and and potentially get some some feedback or at least point people to it um yeah so miguel the floor is yours

Mikhail Kalinin: um thanks tim hey everyone so yeah this eip is um the specification of the ethereum network consensus upgrade to the proof of stake which is driven by the beacon chain in simple words is what we know is the merge and this

specification is the minimal viable version of the merge that we have been discussing for like several months already um like this uh specification uh in its basis has like a number of proposals uh certain from various with alex proposals from the early and not that early days um going through the separation of concerns between the execution and consensus clients and executable beacon chain and quick merge why focus rule proposals also the ideas from the high level design for the execution client that has been presented like a few months ago and the orionism stacks

stack are also in the basis of this uh specification um so that's the huge milestone for uh all the marriage efforts on and for the merch project and the result of the contributing of tons of efforts by different researchers and developers like for more than for about two years um yeah that's like the brief history overview so quickly speaking about this particular specification i'd like to highlight a few things uh in which this specification is um a bit um of a bit uh unusual uh with respect to

other eaps so first of all this um eap um introduces the new hard fork mechanism that will be used for this ap only i guess um it makes uh there is no predefined hard fork block number instead of the block number and the terminal total difficulty will be used and the first block that reached this terminal total difficulty is considered as the terminal fork block and any blocks that are beyond this terminal proof of work block um must be not processed by

the nodes that follow the protocol so for the rational behind this particular mechanism see the security consideration section of this document also the this terminal total difficulty will be communicated by the consensus layer so it's not like predefined it's communicated in in the runtime the other side is that obviously uh the other thing is that obviously we have like like the other side of the logic and specification which is the consensus layer

it's not specified it's just mentioned in several like throughout the document in several references but it's not specified in this eap and um the obstruction that is used for the consensus layer to be like presented in this pack is the proof of stake events uh you may find a section that describes these events they are a bit tricky so be careful with uh reading them um so that's that's how that's the two like things that i was going to mention um also um like

this is uh this is one this is one of the first eap uh relating to the merge the other one at least uh is considered to introduce the deprecation of the difficulty of code in favor of uh duplication of the semantics of the of existing semantics of difficulty of code in favor of this up code to return the randomness accumulated by the beacon chain so we haven't missed this change uh it's just going in the separate ap so yeah that's kind of all i would like

also to hand over it to danny uh probably he has more comments

Danny Ryan: um no i mean i i think you hit the high points uh specifically these events um allow this specification to black box the functionality provided by the beacon chain obviously that spec is handled entirely in these two specs and from a practicality standpoint the functionality will likely be handled from a client on that side um and those events although we don't really bring up like the fact that there'll be a consensus api those events map uh to things that would be coming from that consensus api that we've

worked a bit with in the past um i don't know yeah i think that this uh should read pretty cleanly if you're used to um eips and you're familiar with the inner workings of an eth1 client today so but we'd love any feedback and discussions and you know this is a starting point we'll iterate from there

Martin Holst Swende: um i have a question so obviously the block structure is not changed and we have the hard-coded

cat check lay on empty list and we haven't uh a lot of zeros for the mixers all the stairs for the norms etc how come we're so afraid of modifying the header

Danny Ryan: um i mean this is this is a good discussion point that we can get into uh we're not terribly afraid uh when you do when we do modify those the modifications ripple outward not just from the client but into uh user layer apis and and libraries and things like that so it does probably have a more impact on the

infrastructure providers and isn't as self-contained within consensus so i think that's definitely one of the things i would want to contemplate before um doing deeper changes but i they're on the table we can certainly discuss them and i guess given 1559 and coordination around libraries and infrastructure providers people that have gone through that process here would have a better handle on what such complexities might introduce and how much work it will be for the

ecosystem i think if we don't make some of those changes it's it's a pretty nice user story and that would be i think that's my my primary argument for not making them

Tim Beiko: yeah i guess the counter to that is if we don't do it you're kind of stuck with these fields for you know potentially forever um for as long as we leave them in there

yeah

Danny Ryan: yeah um i guess yeah

Tim Beiko: that's maybe a good question to have with some sort of merge infrastructure called like is it better to have the smallest set of braking changes or is it better to basically you know do

stuff like changing the block uh format then because anyways people are updating their infrastructure and

whatnot

Mikhail Kalinin: changing the block format if we do want to go this route we will have to consider that there are existing applications that potentially can use the block the existing block format to validate this like receipt route and transaction route and even like state state route by by by taking the block header um verifying it with the block hash of

code and parsing it according to the to the box structure that we have now and then using this data in hell so like if we drop some fields at the end of the block header it might not break um these applications but anyway we should consider this right

Martin Holst Swende: you mean like if well some yeah on chain validation of previous headers uh it might be worth yeah looking into how how much of those exist and how they're written

right

Danny Ryan: that's kind of like well not quite the same but uh any sort of validations against block against the hash and the state root um are likely to be broken at some point and hopefully there's not a ton of that on

chain you guys

Łukasz Rozmej: so we were discussing in the context of london potential bugs issues and like rollbacks and um etc so i'm not sure if there is anything we can anticipate with going to proof of stake that we should

might roll back to proof of work for example but again modifying more things like block headers etc would make this even more complicated

Danny Ryan: right that's fair and i do think that's a decent amount of work to do over the next few months is better i think the security considerations and different paths available or in a number of our heads but we want to spend a lot of time writing that down and working through contingencies and that kind of stuff as you can see further threat analysis

at the bottom is

tpd

Tim Beiko: any other i guess questions or comments on the ip i thought it was a discussion about mining in the comments

Martin Holst Swende: um uh yeah well i i just like to follow up on that last question one more thing so with the there is the concept of finalization right so if

the majority each one client produces and what you know an invalid or incorrect state route and it gets accepted and eventually finalized so what would be the what what's the timeline between that invalid state being included and it being finalized and after that would that be in any way possible to roll back finalized state or do we say that no final life finalized

Danny Ryan: so finality happens in the normal case in two epochs which is about 12 and a half minutes um and there's a couple of things there

so if a majority client or set of clients that represented a two-thirds majority did have some invalid state transition that they agreed upon and they were able to hit those thresholds they could finalize such a state but you end up having a pretty similar thing as in proof of work if i'm running a client that does not agree with that state transition i'm just now going to be on a minority chain that is being built but it's not necessarily being finalized and at the end of the day you can specify uh you know you can manually override uh and certain command-line parameters should probably

exist to be able to manually override such that uh even if something was finalized finalizes uh you know there's something social there but it's also like a practicality standpoint like that unless manual intervention is final and will remain within my chain but in the event of of error you can always intervene

go on

Martin Holst Swende: yeah i was just i think that social element is going to be pretty large because i think the you know the presence of the word finalized right to a lot of users they don't care about the technicalities they you know oh it does finalization you know

yeah i i see that i see that um and maybe we should consider a bit more what that path is i mean it's from it's a yeah um okay you know practically something 100 blocks deep and proof of work is finalized as well but uh it doesn't carry the same term

Micah Zoltu: i don't think so because you can you can have a shadow fork that will cause clients to automatically switch

after i agree

Danny Ryan: i agree it's finalized from the way that people generally handle it and think about the state there but it is

subject to more uh attack vectors than a proof-of-stake finalized chain would be

Tim Beiko: um i guess just because we're almost on time um what's the best way to continue this conversation i guess it's the there's a discussion to field in the eep um we obviously have the discord for more like synchronous stuff um anything else miguel danny that you think where you think people should kind of keep track of and and leave their comments

Danny Ryan: i would say those two there's a merged channel in the discord that we can specifically talk about obviously a lot of this might ripple out into awkward devs at a certain point um and the discussions too is fine mikael is there anything else um

just

Mikhail Kalinin: uh yeah about the calls that are devoted for specific topics so if we will have something big to discuss like consensus api that involves uh both um like beacon chain and may need client developers i think it makes sense to make a call devoted for this this kind of discussion uh which will be just announced in advance and yeah and will happen on demand

Danny Ryan: and mikhail you're also implying that we are likely going to deprecate the merge specific call that was regular and move to discussions in all core devs and the eth2 specific call and then if we need kind of a breakout room call on a specific topic

we'll do that on on demand right

Mikhail Kalinin: yeah right so and yeah we we are temptatively booked uh this slot that previously used for the marriage calls um for this kind of breakout sessions

great cool

Tim Beiko: um we only have three minutes and uh yeah light client has pointed out a couple times in the chat we kind of got lost in the bike showing of the names earlier um but didn't actually get like a super strong commitment on the actual next step and what we want to focus on um so it seemed to me like the most people i think basically everyone aside from potentially aragon really wanted to focus on the merge next

um that if we kind of got to a spot where we have merged test nets that might be a good spot to reevaluate um what we can put in shanghai or whatever we call what happens in december um does anyone disagree with that to basically you know kind of focus on the merge starting from here get to a spot where we do have you know ill-implemented in all the clients and it's running on test nets um and depending how you know how much bandwidth we have there um we can possibly do some features for uh the december upgrade but if we get there and realize that you know we just don't have the bandwidth then it'll have to move uh further out does that generally make sense to people

are there any strong objections okay uh no objection so i guess yeah at a high level um let's do that um unfortunately so there were two eips more that people wanted to present on this call um 3670 and 36.51 um 3670 was from axic um and i know william you kind of sat through this entire call um and you're not you know kind of a necessarily regular attendee so do you want to take like a minute or two to just quickly present your eip and and

point people to where um yeah they can they can uh they can interact

William Morriss: i've shared my uh prepared statement on ethereum magicians and posted the link in the chat

Tim Beiko: okay thank you very much um cool and then uh there were yeah uh so next week at basically alkordev's time so exactly one week from when this call started we're gonna have uh 1559 and london uh called for infrastructure providers so if you are working on an application wallet or any other piece of infrastructure and are in the process of having 15 implementing london support and have

some questions um please come then and we can try to help you out as much as possible um and lastly uh there's two uh peep and ep sessions happening that are pretty relevant next week um puja do you want to take 30 seconds and uh yeah go over those two

Pooja Ranjan: oh yeah uh so uh to celebrate the ethereum's sixth anniversary we are organizing two special people one with eip 3675 the update consensus that mikhail just talked about and that is scheduled on july 27th at 1400 utc the another one is going to be um that is not a proposal specific but a topic that people are looking to learn about that is blockcast limit with

vitalik butarian on july 30th at 1400 utc um we are also organizing one call for specifically for the london upgrade that is carried on 2nd of august at 1400 utc where we are inviting all the ap authors the client teams and the infrastructure implementers of those proposals of london so please join it and gather this thank you so much

cool

Tim Beiko: any other final comments thoughts from anyone okay well thank you very much and see you all in two weeks

oh and upgrade your nodes for london um so and keep an eye out for uh the ethereum foundation blog post uh today or uh or monday uh with the release versions um but yes london will be happening before the next awkward f's thank you then cool thank you thanks everyone thanks everyone bye thanks

bye [Music]

[Music] [Music] [Music] [Music]

[Applause] [Music] [Music] [Music]

[Music] [Music] so

[Music] [Music] [Music] do [Music]

[Music] [Music] [Music] so [Music] [Music]

[Music] so [Music] so [Music] [Music]

[Applause] [Music] [Music] [Music]

[Music] [Music] foreign

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.