All Core Devs - Execution #136

Transcript

[Music] [Music] [Music] [Music]

so [Music] [Music] so

Tim Beiko: good morning everyone welcome to awkwardev136 i appreciate people coming on the call on what is a holiday in a lot of places um so yeah thanks for coming in hopefully we can wrap this up uh within time uh basically

a lot of a lot of merged stuff on the agenda today and um a couple shanghai updates mostly on stuff that's already been included um and yeah to start on the main net stuff we did have or on the merge stuff sorry we did have our first mainnet shadow fork uh this week unfortunately both perry and danny are in transit def connect um so they can't give the high level summary um but i'll i'll try and then it'll be helpful to get client teams to chime in um but as i understand it the mainnet shadow fork uh had some issues across across a bunch of clients but we did manage to finalize uh on on

the other side of it uh on the shadow fork so um you know from our like high-level network point of view it it went well but it obviously uncovered a bunch of uh smaller issues across uh several clients which uh we we then need to fix um we're planning to have a gordy shadow fork next tuesday and another mainnet shadow fork next saturday perry's posted the configs for all of these in the discord already um and then for the mainnet one uh obviously if folks want to participate in that you need to sync your main net node which might take a couple days so just be mindful of that um but yeah i don't know if any of the client teams kind of want to just share

some of the issues they they ran into uh during during the shuttle

fork

Marek Moraczyński: uh i can start uh so we had problems during uh the transition most of our notes expect nimbus are recovered now and they are working fine on maintenance shadow fork uh uh i would say that the main reason was that our sync is it's not ready and we are aware of it but of course we don't want to miss any uh merge testing so uh even if we are not fully uh ready we we are trying to participate in every marriage love that

i hope that in new shadow fork we won't have the same problem during the transition but on the other hand i have to say that we are still working on sync to give you more details it was connected to peer refreshing we had wrong peer information and also we had to run condition in our beacon header sync and because of that the sync was stuck about if stats uh it is probably only reporting to if stats back because i don't see delay when i'm observing nodes they are just producing uh blocks and uh sorry uh processing blocks in uh correct

uh way so yeah that is all from my side

Tim Beiko: thank you for sharing um anyone else

Justin Florentine: hello this is baesu here um we had a couple of issues uh with uh with the shadow fork um a couple of more configuration issues that were quickly sorted out um two of those were concerning and are being worked actively right now one of them was for getting uh syncing we had an issue where we had a fast sync that pivoted across ttd and ended up using the incorrect validation rules uh and that's that's been addressed there's been a whole lot of uh attention being uh spent on our synchronization

um so there's a number of pr's in flight for that uh the more concerning one is still being investigated right now we had a receipt mismatch um that would be a consensus break on our side we stopped validating and we're basically hung up maybe you know 10 or 15 blocks after ttd hit on the shadow fork so we're still investigating it um we're pulling traces and uh we're comparing them and trying to figure out how that happened

Martin Holst Swende: so what was that just one node that displayed that or was it something which was experienced across all nodes they

received

Justin Florentine: so yeah so there's only two nodes running and they um we expect both of them to have the receipts roots mismatch problem

the other one just didn't hit it for a synchronization issue

Martin Holst Swende: but do you not when you download the blocks and you're saying do you not verify their secrets

for the bodies

Justin Florentine: no we do verify them and unfortunately we came up with a different answer so we treated it as a bad block

Martin Holst Swende: but how could they not both hit the issue

of them

Justin Florentine: oh because the other one didn't get that far it was hung up for a synchronization issue later or yeah earlier

Tim Beiko: right so your only node that got to that point got the received mismatch but yeah if you had a second one you think obviously you can't know for sure because you haven't found the issue but you think it's likely it would also hit the same thing right

Gary Schulte: i believe we did have a non uh ef node that had the exact same receipt root issue got it it's just that the two nodes were talking about were the ones that party were sorting got it

Mikhail Kalinin: and i guess it's not related to pre-render good because there were no such transactions submitted on the

mainnet

Justin Florentine: you know that's the first thing that i looked for and i didn't find any evidence of that so yeah

Tim Beiko: got it um yeah thanks for sharing and keep us posted uh once you find out what the issue was um

aragon guess anything on your end

Martin Holst Swende: um i might be the only one guest team here and i can't say a lot i've been i'm gonna we're on vacation for a week um as the week before that i spent trying to to find issues that were uncovered in when we did the the garlic shadow fork uh two nodes some i got corrupted state data or snapshot data and we were we have still not yeah with 100 certainty rooted out what the cost for that is so um yeah i'm looking forward to more

more of these shadow forks and girls out of work to see if we can reprove it again that's it

Tim Beiko: got it andrew

Andrew Ashikhmin: yeah i think i i haven't been heavily involved in the shadow of work testing but i think it went uh all right one issue though um the i think both lighthouse and prism uh call net methods on the engine api port but we we only expose each methods not net methods so i would like to clarify can can lighthouse and tourism stop calling

net methods on the engine api port uh or otherwise let's standardize it because in this pack it only mentions it's message so i'm not right now i would rather not expose net methods on the engine api port anybody from

white house prisma

Terence Tsao: hey um terence here from prism so yeah i think that's a bug that we will fix i don't see why we have to use snap method i think everything could be under the everything could be under the if method so yeah we'll fix that that's let's let's bring that to my attention

Andrew Ashikhmin: okay

thank you and i don't think we have anyone from lighthouse is that right

Mikhail Kalinin: yeah but we don't need net methods on the engine api board definitely still it should be removed uh as i understand that method which requests the version of network which is network id in each status message so it's like if each chain id should be a good replacement of that or as uh adrian from taco posts in this court we can use to

exchange transition configuration method to verify that we are on the right screen because terminal total difficulty set for the mainnet for example will will never be messed up with any

test net

Gary Schulte: is that a an execution spec change to include network id chain id on the uh on that engine api or is that uh is that already in there

Mikhail Kalinin: uh it's already in the spec if chain id and we will not add

any any method from that namespace yes so yellow client that follows the stack should expose all the e flat that's included in the chain 80.

Gary Schulte: uh right i thought we were talking about adding that to exchange configuration exchange transition

configuration

Tim Beiko: um i see yeah we have at least uh prismatic here um yeah do you want to give an

update from the the cl side how the shadow fork went for you

Terence Tsao: um i can give an update on the prism side yeah and i think everything went relatively well we discovered a few of edge cases but there are no consensus breaking so people doesn't necessarily see it so there's definitely like things that we should improve on in terms of performance and then and then we are also working on the ux as well because i i'm monitoring out these four channels and there's like a lot of people that don't necessarily know like still don't know how to launch a node between cl and el so we're working on the we're working on the uh side to make it more uh smoother for the end users and yeah that's pretty much it

Tim Beiko: awesome thanks anyone else from one of the consensus layer client teams sorry i don't recognize the names as easily okay i guess not um so yeah thanks thanks everyone for for sharing these and like i said earlier like we'll have two more shadow forks next week um which you know hopefully give us uh no new issues and uh if some of the current issues we have are fixed uh we can we can test them in production um next thing on the agenda uh mikael you had you've spent some more time this

week digging into the latest valid hash uh you you shared your analysis on on the agenda but maybe it's worth it for you to take just like two three minutes to walk us through uh your thoughts and uh yeah the hackmd that just was shared in the in the chat you can either share your screen or just talk whatever uh works best

Mikhail Kalinin: um yeah okay so yeah okay i'll share this screen thanks then can you see my screen

no yes okay so cool uh a brief background on it so we have latest validation the engine api is back and uh basically what the stack says is that while el client is sinking uh it may encounter in invalid payload invalid block and since cl is since we have like unidirectional uh communication channel where cl posts send a message method call and get a response uh el will have to store uh an information about invalid chain um to later respond with this information

if uh cl requests and do any do any method call uh with the information from this value chain so that's actually what's stated in the spec and yeah uh this discussion started from uh like yeah this discussion started because apparently it's currently currently el clients doesn't store required information some of your clients do store uh a few recent invalid blocks but that's not enough to serve this data correctly um so here is the like implementation how it could be implemented uh like

a kind of implementation tips and then we just uh we're wondering uh whether we can't just admit supporting this during uh el clientes thinking and i was just you know uh thinking about it more about possible attack scenarios and this this attack scenario seems like uh it has it may have a big impact on the network and it's its likelihood is not that you know it's not that high uh it may happen during the period of no finality uh where this period lasts for like more blocks than uh we have a state sport so if like period of no finality lasts for more than

uh 128 blocks and the clients doesn't have a doesn't have a state to execute a submitted block if it branched from the like canonical chain more than this number of blocks uh which is like which which which may occur uh so we are risking to have uh all all validators which will try to process this block in an infinite loop of trying to sync with incorrect within the valid chain so so you can align this failure uh which will require manual integration yeah this this is my these are my conclusions so you may take

a look um yeah you may just take a look at this uh and based on this uh analysis based on this attack scenarios i can conclude that we have to support uh have to support this functionality as stated currently in the stack so we will likely have a uh test scenarios that will discover this kind of stuff so that's that's all from my end on this topic happy to answer any questions

um i

Andrew Ashikhmin: i have a question so

if a block is missing so the el tries to download a block but um then it can't and it times out um we first like we so if there is a gap on the side chain that we don't know the exact latest valid ancestor and also that timeout perhaps it's transitory it's it's difficult in that if you kind of you you you have you've downloaded a blog and you've uh you you've found out that um something is wrong with it with that block then it's fine you you

uh you can go through uh through the ancestors and uh find the uh the last valid one

Mikhail Kalinin: but if you have a gap then kind of a bit lost uh if you have a gap you will not be able to uh oh you mean okay so if if if like you're yeah see so it may be that like first block in this fork and this side side side chain this side fork will have will be invalid right then then you have a gap you missed your missing second block and then you have a you have third block and the consensus

uh the client just submits like the fourth block it's this kind of situation

Andrew Ashikhmin: i mean by gap i mean you like uh you don't have uh you you you the el doesn't have the missing blocks locally it has to download them but the download from from its peers but the download fails right

Mikhail Kalinin: yeah because because it's invalid right

Andrew Ashikhmin: yes for instance because it's invalid or but then so one question is it might fail because it's embedded on might be the network is temporarily down right and then at some point it

the connectivity is restored

Mikhail Kalinin: yes and uh in this case yeah that's interesting case because if you have this kind of issue uh like so it's like a data availability problem on the outside right i mean we will be in the data availability uh data unavailability issue right we'll have it

Andrew Ashikhmin: so this is what we mean because there is so first this data availability issue so that you can i guess never be certain or you you might approach certainty but not

be 100 sure that whether it's a data availability issue or the block is invented yes and on the other the the second problem is that if you don't have uh the block then you cannot establish the latest valid

um ancestor

Mikhail Kalinin: uh okay so the first data availability problem is where uh will not be should not be an issue on the outside because in this case you will be what we are like what this attack scenario is about there are online nodes that are

that have been synced with the network and they are importing uh received blocks from the network in a lockstep mode and there is the possibility to make them um like to turn them into this infinite syncing loop with invalid chain but yeah but they will receive all the beacon blocks uh from the malicious chain before so they will there will be no data availability issue in this attack so it's like you know when you're fully seen uh data availability becomes the responsibility of uh consensus layer and if you don't have a beacon block so you will not you will never

follow this chain so yeah there is an assumption uh that we have all blocks here and we can't exactly understand which one is invalid

Andrew Ashikhmin: uh okay cool thank you

Mikhail Kalinin: yeah but yeah the case what you what you are describing may may happen while node is sinking and it pulls blocks from uh execution layer network i was also like a kind of thinking of this scenario and then just understood that it's not uh that online notes uh that are fully synced before they just not affected and yeah it's okay and they will just drive all these sinking notes through this period of instability

and yeah they will just rework uh this is the yeah this is the second attack you know so with the sinking yell yeah this yeah second scenario which is not that impactful uh like this first one

Martin Holst Swende: so what just about this first attack what is this special thing about you said 128

blocks

Mikhail Kalinin: uh that's just you know this because the number yeah this is where uh this is the default number of uh most recent state versions that clients are storing right that get for example yeah so that's that's about it yeah yeah so that's that's the uh this

attack may occur when uh adversaries submits a block uh which is invalid but you don't have a state anymore to execute uh this block and uh execution where a client will have to pull it from from the network uh yeah

Martin Holst Swende: and we'll so if it's the case in that particular case if if the state availability thing is is what you're thinking about i don't think so we at least forget we still have the block we just don't have the state so we just have to go back somewhere where we do have the state and then regenerate the state but it's not a something we need to put from the

network in this case

Mikhail Kalinin: okay so yeah but this is like uh this state miss missing state issues just uh because we need adversary i would like to make the uh notes and into sinking state that's it and then in during sync they will just encounter that this payload is invalid and if they just drop this chain and never say back to cl that there is an invalid payload just please invalidate this chain and never send me again

uh ciel will uh try to to to stay on this chain i'm trying to induce the sync once again and so forth and so forth so el must report back in this case i i like just assume that this common ancestor with a canonical chain and this common ancestor uh yeah four state of this common ancestor is the requirement to execute this uh first invalid payload and the uh uh yeah the condition for this attack to happen that we don't have this prostate it's been proven and yes el uh have to pull this uh

state first and to make uh to to to pull the state that induces uh scene process and response thinking uh that's that's the scenario

Tim Beiko: um martin did that make sense is there anything you wanted

dad

Martin Holst Swende: i

don't have i can't say that i fully understand totally everything about this attack but i don't have any further questions yeah

Tim Beiko: fair enough um anyone else yeah looks like someone came off mute and went back

Gary Schulte: on i was just wondering what what's the what's the size of i mean uh do we just need one malicious proposer in this case or how how coordinated would this type of attack need to be

Mikhail Kalinin: yeah that's a good question we may have like a couple of proposals i guess because in the proof-of-stake uh yeah and we also yeah need some

um we need the ability of we need enough of steak to to game to game with lmd ghost uh fork choice rule so you have you um yeah that's also a condition like to make this attack happen if adversary owns an amount of stake that make uh cl clients uh which are staying on the canonical chain somehow to rework to do this like malicious chain yeah probably it's yeah probably it's invisible uh taken in account that there is the 128 blocks uh in the canonical chain already uh to to make this happen

but we also have like shallow state clients so there yeah this probably they like the likelihood of this attack it's like pretty low but yeah i would just want to prevent this this one anyway and like oh sorry go ahead yeah go ahead

Tim Beiko: i was gonna say like for uh by shadowstate client i think like aragon is probably the main example here like i'm curious to understand just like just from uh engineering perspective like is it feasible because i i know we changed kind of the requirements around the spec because it was really hard for

uh for aragon and maybe another client to like support it as is but like yeah how i i'd be cursed to just hear you know like how complex is it to like mitigate this issue

Andrew Ashikhmin: and yeah i've already started i've introduced a kind of a basic level of protection this cache of invalid blocks but it might be not bulletproof yet so i'll revisit the i'll review the michael's attack scenarios and i'll revisit aragon's code but uh

i think we should be able to mitigate against such

attacks

Tim Beiko: okay awesome um it does feel like i know my like you know people need time to digest this and and think through it a bit more so we should definitely discuss this um i mean both in person next week with whoever's there but um yeah on the next call uh based on you know people having having spent more time digesting it

Mikhail Kalinin: yeah it would be great yeah if you take a look um if more engineering eyes will be on it probably i'm like overestimating the

Tim Beiko: uh yeah i think one thing that

yeah beyond just like uh el engineers also having folks like from the proof of stake side like if this does require you know say a third of this stake or two-thirds of the stake um are there i guess it's like are there worse attacks you could already do with one third of the steak if that's what you require here

um yeah that's fair yeah

Tim Beiko: but yeah i think just like from those two angles if we can look from like how easy it is to mitigate and then is this actually the worst thing or like one of the worst things you can do with that level of stake that can probably inform us about how how much you prioritize fixing this and changing the spec any other questions

thoughts about this okay uh thanks mikael for for putting this together and sharing it with us thank you um cool so the next thing uh i wanted to make sure we cover so on on the last alcor devs i think we were short on time and we like briefly touched on like test nets and i know there were like some differing opinions about like how we might want to approach rolling out on test nets obviously once like the shadow forks have stabilized and we're in a better spot generally um but i think you know there were some comments about like maybe we should uh spend more time on testnets than we usually do

um and and and trying to figure out like what's what's what's the sweet spot and and what uh and and you know based on that i think that kind of affects you know how the the time it takes to go from the first test not to maintenance um marek you shared some comments in the agenda uh do you want to take like a minute or two to just go

over those

Marek Moraczyński: uh yeah sure uh so we had a conversation with team why we think that it is it will be valuable to have longer test net and we discussed a few risk one of them is obvious client backs and i believe that the sooner we start working public tests the longer time we should observe it

and there is tomorrow let me start with i think less important so all our devnets are controlled by our wizard hero paritos so it seems to be big step if we move to network from network to control by controlled by one person to controlled by many people and team proposed that maybe we should try staffing closer to public test net in terms of validators control and i don't know what do you all think about it uh should we do it or should we try it on a real test

net like sepal yeah what do you think

Martin Holst Swende: so one thing i think immediately is that it might be difficult to get people um to actually commit to actually go through with this and in the end we'll just be um yeah the same team so the the develop the dev teams that are doing it

right

Tim Beiko: but i think right now what america saying is like uh perry is running like most of the nodes even on behalf of a lot of decline teams so like i'm not sure how many and i might be wrong here but

i'm not sure how many of the actual client teams are like setting you up and running their nodes with all the say all the els running with each cl and vice versa um on the current shadow forks yeah

Marek Moraczyński: i think client teams are running nodes but the still finalization is all

uh depends on parry nodes so got it

Marek Moraczyński: yeah so uh yeah network uh will work without client team nodes but not work without

parry nodes

Martin Holst Swende: but how would you rather see

it then

[Music]

Marek Moraczyński: so maybe it was like a proposition only uh if we should uh do something here so what what my question is a bit who will control the test net uh notes that's not validators

Tim Beiko: right yeah yeah we i don't think we ever agreed to like specifically who but there was we we wanted to have like two basically two test nets one which is controlled mostly by the ef and client teams um so it's like a bit more stable and maybe some infrastructure providers uh and then the other one which has a bit more of an open validator set uh

where anybody can join and on that one you might expect some uh you know some non-finalization and things like that wouldn't it oh sorry go ahead uh no no no no fork sorry gary you kind of broke up oh if you're speaking gary we can't hear you anymore okay uh well if you get your audio back

uh gary please chime in so i guess you know we do have two shadow forks coming uh one tuesday and it's probably a bit late that the change cleans up too much on that one but we have one also scheduled for like next saturday um would it be helpful to distribute more of the validators to client teams so that finalization doesn't depend just on on perry um is that like too quick to do it would we want to do some like maybe in the week after that which are set up that way

Marek Moraczyński: um i think we can talk about it with barry

because it he will be the best person for that uh i i'm not sure if the notes are in internal networks so uh barry should uh say about it more uh okay uh so maybe the second point uh so miss configuration um so in every hardware we observe some amounts of notes that were forgotten to upgrade and all previous hardwares were significantly less demanding for note runners and it was just download the new version of the client

now note operator need to do a few things uh upgrade cl upgrade el configure uh secret between clients make sure that there is connection between clients so in theory people can train it on killing and we can mitigate it with correct announcements and documentation however in practice i believe that the longer test net the longer public testing will reduce the number of misconfigured nodes for the next testnet and later for for mynet people need to get used to running nodes uh with two applications and that was my arguments for longer tests

yeah and that's still from my site team do you want to add something

Tim Beiko: i have thoughts but anyone else from other client teams want to chime in first if not um i i agree with you on the configuration issue i i do think like this is kind of a higher uh or more complex change than like we've we've had with previous upgrades uh and hopefully kel has helped a little bit move the needle but clearly a lot of people and the vast majority of people are not on kiln um so i guess it does seem like you know you probably

want to give people more of a heads up for at least the first one so that they get the configuration right but then once they've done it i think you can probably you know assume that they can do it right the next time again assuming things things go smoothly on the fork um so my view there it's almost like you would want to separate like you know if we're gonna fork three test nuts like if we're gonna fork sepolia gordy and robsten um you might want to like have the first one happen and leave it running for a bit long like give a bit more of a heads up for people to configure those nodes leave it run it for a bit longer but then you know as soon as like uh the fork happens and and and if you if people have configured things correctly and it

didn't blow up then like you can probably have the two other ones you know fairly close apart like i don't think um because people are gonna have to run through the process a few more times but it's like the first time will definitely be the hardest um so my general approach would probably be like fork something like robsted first because it does have like pretty significant usage um but we don't intend to keep it around much so if if it doesn't go super smoothly um not the end of the world um once you know robson is is forked and like stable then like you then schedule kind of the forks for say gordy and zipolia and then you also kind of get that say

you know say you like fork robsten and like two three weeks after that before gorty or sepolia by the time you're forking gordy and then sepolia um you've you've already had robson be live for a month so i think you get this kind of longer uh duration on a test net and we can do things like obviously syncing new nodes make sure that still works uh robsten has a pretty big state in history so it's it's you know it's a good kind of proxy um so and then obviously we'll want to see all the test nets stable before we move to mainnet but like it would probably look something like i don't know robson has had like a month a month and a half and then like maybe you do gorty second because there's a ton of activity on that as

well um and you get like longer data on that and then maybe sepolia will only have forked for a few weeks uh but if we have like kind of long-term data on on robsten medium term on gordy then that's probably sufficient um and just i guess they bring it back that also one thing you said at the very beginning you know like the the the earlier in um the earlier in the process that we fork test nets the longer we want to see them run fine but i do think like with the amount of shadow forks we're doing and we're going to keep doing my expectation is like we would only move to test nets once the shadow forks out of say main net are going really

smoothly so like to me it's like if we're if we manage to for the shadow fork main net smoothly the test nets should be kind of a easier than that um yeah so i don't know like i guess yeah the recaps like i would probably do robson first wait wait a while make sure it works then probably bundle gordy and sepolia um and then once that's all set you go to mainnet um and then martin had a question in the chat uh about uh paired with cl test nets uh the answer is no the the only one so gordy has a long-lived cl test net um which we'll kind of transition after but for robsten for for sapolia we'd have to to launch new cl test nets yeah andrew

Andrew Ashikhmin: um yeah can we also agree that before we [Music] migrate a public test net we abandoned the uh on on unauthenticated port so we leave only the jwt port that would be i think eight five five one by default right so we leave only that port if that's all right with everybody

um [Music] wouldn't that be kind of an implementation choice

Andrew Ashikhmin: uh i think uh we wanted for security reason not to like because that the uh unauthenticated port was only for com kind of for ease

of testing rather than working solution

all right

Martin Holst Swende: so if i understand you what you're proposing is that we should not serve engine under the unauthenticated

Andrew Ashikhmin: yeah to do to disable to not to serve the unauthenticated port before um before switching public

test nets

Martin Holst Swende: but are you proposing that rpc services should not be available under the legacy support or you're proposing that the engine api should not be exposed

Andrew Ashikhmin: uh that well

in aragon how we have it we have three ports uh or even four maybe so one for like non-engine api port for ether and everything else and for engine api there is uh there are two ports eight five five zero unauthenticated and eight five five one uh jbt authenticated so i proposed to stop solving completely the unauthenticated uh engine api port right okay yes then i'm with you and i nothing against it

Tim Beiko: yeah and it does seem like especially based on the configuration stuff we were just talking about we definitely want to force people if if on mainnet the engine api will only be available through an authenticated port then we want to force people to make sure that works on the very first test net because if it breaks for some reason we want to know then and not when we're forking maintenance any other thoughts comments just on test nets generally okay so i guess to recap just to make

sure there's no objection like moving with robson first um giving people a bit more heads up so that they can set up their their nodes um making sure that goes goes smoothly and then once it has then going through gourley and then sepolia at the end so that we get kind of the most data on robson then gordy then a bit less on zippolia gary

Gary Schulte: yeah i was thinking um it seems to me that sympolia might be a safer target given that there's not as much infrastructure there uh if we we wanted to migrate to polio first but for example i don't think that there's uh ether scan support for so polio yet i mean we're basically going to force

um robston users to have a consensus client set up in order to continue to participate and i think since depolis really mostly only i've only ever seen you know like uh you know core devs and folks on that network i think it might be an easier target for the first migration

Tim Beiko: right it depends what you're optimizing for right like so if you're optimizing for the fork going smoothly i i agree but i think merrick's comments was like if you want to optimize for people already running nodes having to figure out how to set up an el and cl combo um then having them do it on sepolia like people might just not do it on sepolia so that's just the risk is like say we

did sepolia robson gordy then sepelia goes well but then on robson everything breaks because like the node operators and the folks like say etherscan or like exchanges or infra providers that just support robson they they just didn't do anything on sepolia um that might slow us down more so i yeah

Gary Schulte: yeah that makes sense i had an internet cut out uh when when merrick was talking so i missed the point that we're optimizing for breakage but yeah

Tim Beiko: so basically i think it's optimizing for forcing people to to think through the configurations and make sure that they work um and then that basically leaves us with like gourley and robsten and of those two i

think breaking ropsten is less worse because we don't expect to support it long term after the merge so i don't think we should go ahead and break gourley even though it probably has the most usage so it's like robsten is like a a medium test run and then if that goes well we can move to gourley and then sepolia should be really easy to do after after those

because that makes sense

Tim Beiko: cool and i guess as a heads up uh this is kind of implicit but i'll share this uh explicitly uh just to be clear we had discussed a few times that like rinkeby won't run through the merge um so just for people listening um

this ring can be will not uh i don't think the network will be like shut down right at the merge but from as soon as the merge happened rinky just stops being a replica of of what's running on main net so people should be aware of that and start migrating away from ranking anything else on

test nets any thing

Tim Beiko: else about uh the merge gener generally

okay uh so next thing we had was uh a couple things related to shanghai so we already have accepted the eof eip in shanghai so eip 3540 and there were two proposals about how we might be able to to tweak i guess eof to either ban self-destruct or uh later groundwork for uh for code chunking for vertical tries um i know andrew you had uh left a couple comments about this uh do you want to maybe just give some context here on you know what your what your thinking is um yeah

Andrew Ashikhmin: um i think uh

eric wants team preferences to disable self-destruct instant height but if we don't do that if we do it later then my initial thinking then maybe we should disable it in for an eof code but then uh exec has made a good point that maybe it's counterproductive because if say we change self-destruct and to send all some something useful uh then it doesn't make sense to disable it in your af code so i think uh i revoke my suggestion i i don't think it

makes sense to disable self-destruct specifically for eof code yeah and the second point is about code chunking um i think code chunking is kind of it looks rather hairy the code chunking for for the vocal tree right so and it kind of looks ugly because then when you e you contaminate evm code with

vocal tree commitment logic so kind of maybe i i'm hoping for a more elegant solution i don't have any good ideas but if somebody comes up with a more elegant solution for code chunking that would be terrific uh my concern is that let's try to think carefully about eof and code chunking so that the uf changes that we introduce in shanghai don't make don't make the introduction of the vocal tree harder specifically in the area of code chunking

so again i don't have any ideas i just want people to think carefully that we don't um yeah introduce changes in shanghai that would make the vocal trees more difficult

could

Martin Holst Swende: do you have any any like more concrete example of what you think is yeah what part of it is bothering you

Andrew Ashikhmin: well because eof it's um with the before uaf uh all co there was no structure in the code and the only uh difficulty was to differentiate push data versus non-push date right so but

basically it was just a string without no no structure and you can chunking you could count the chunks of that string with eof you introduce some extra structure right and uh uh so you you have sections with the uaf how do those sections align with chunks right that's that's my worry

okay yeah uh powell you have your hand up for a while though

Paweł Bylica: yeah actually i i wanted to go back to the uh to this service track but i can start with this this chunking

um so we were thinking about that a bit in context of eof so that there is one byte wasted uh in the in the chunks currently to encode the the self-destruct uh no the the the dumpling locations um so we were thinking how to actually if we can mitigate that with eof and one of the ideas was to have separate section that encodes the jump dislocations outside of the code so we kind of have mixed feelings about that uh there was some some working

into this direction but uh it wasn't so so much smooth in the sense that encoding that in a separate part of that i as i remember it wasn't like so great that we wanted to push it as a first thing um and kind of i like i'm personally thinking about that for the future that they might be a way to to remove the jumps as they work currently with something else that will not need to have the jumped at all that would kind of solve the problem

uh but there's also nothing that is ready as of now so i think it kind of also depends of the timings uh i think in the case we would have that ready the specification for some such features ready before the immigration migration to the workers happens then we can consider that but as as of now it looks it would be some future version of eof uh because we don't ship with that straight away

Tim Beiko: uh you also wanted to say something

about self-destruct so maybe

Paweł Bylica: yeah yeah i have two comments to that so um well first i think uh the the the most important question is if this sent all is is useful in the things that we want to have it or is just the the minimal way not to break a lot of existing contracts so i'm not sure like what's what's your opinion about that uh but like technically um uh like disabled safe distract during eof validation is is very easy to do and that also means the

the evm will not have to be changed i mean it will not to have to modify how self-destruct works uh because it will just this instruction will just not show up in the eof code so um yeah it will not be just you will not be able to physically physically place that instruction in the code so we don't really have to switch evm in the sense that it will have to different behaviors for eof and different behaviors for whatever the self-destruct for legacy code will work um so that's one thing and secondly

um eof has some let's say forward compatibility properties and even if we start when it's it's it's much better to start with disabled self-destruct because later we can actually assign even if we decide to like enable send all in in this op code that will not break previously deployed uf contracts so like one of the properties of this forward compatibility is that you can safely assign unassigned opcodes

so let's say we if we start with self-destruct of code unassigned in the in the eof then it's it's easy uh to enable it later um yeah that obviously that that requires hard work uh but we have guarantees that will not break

anything

Tim Beiko: got it um alex you have your hands up as well

yeah i had a uh something to say about the the world retreat chunking uh even before the uf we actually worked on uh

with cena uh we worked on like an earlier code checking proposal and i think the the vertical three one still in essence works similarly uh because it has like this first extraction offset byte and so when we we started working on uf we actually kept this code chunking in mind and compatibility with it and eof is compatible with the historical three spec the only the only noteworthy thing is that the eof headers they're small so they always fit a single chunk but in practice uh for uf contracts uh you would need to always provide the

first code chunk as part of the proof because that's where the headers are situated um and in fact when we had a decision to be made whether we uh have the the entire eof headers upfront or whether we have like a streaming version this was one of the reasons we decided to have it up front because that would be a single code chunk in terms of code chunking and secondly while it works without any issues with the current um vocabulary spec uh it could be made more optimal if if the oracle 3 spec is aware of eof then you could potentially remove this first extraction

offset in case of uf contracts and you would say one byte per chunk but i mean it doesn't need to be done it would be just an optimization

got it thanks uh andrew yeah it's

Andrew Ashikhmin: great that you have that compatibility in mind i guess that was my primary worry um also about i have a question in eof in general like other incentives for people to move to eof any kind of gas reductions anything like that

um

well not not directly in the not not on its own but we do have a a bunch of other proposals um which are not yet slated for shanghai but the the biggest one uh the most relevant one for now would be the the static relative jumps which i think is is would be ready to be considered for shanghai but obviously the discussion is now paused um but that is only possible to be introduced with the uf because it depends on immediate values and that reduces gas cost regarding control flows significantly because at least looking at solidity contracts which are the majority of the

contracts on chain over like 90 percent of the jumps are static jumps in a solid contract and basically it would benefit from the static relative jump code quite significantly we don't have um actual concrete numbers about the reduction but we do plan to um to create some statistics about uh some major contracts like unit swap uh typical comparison between you know the current gas runtime gas cost or like a swap versus um after using this uh with static competitive chance thanks

Tim Beiko: anything else on eof

either the self-destruct vertical tries there

Greg Colvin: excuse me i still can't find the hand on here just just in general eof lets us make a break and say we're moving forward with some new stuff and we need some way to say this is the new code and to be able to say these old codes no longer work so it's just a break we need to make going forward getting immediate data is the most important thing right now push push is the only thing that has anything resembling immediate data

so it's we just have to have it um going forward that's all

Tim Beiko: thanks for sharing anyone else on eof

Andrew Ashikhmin: um yeah just um uh i kind of i'm still on the uh on the fence uh regarding disabling self-destruct and and say cold cold uh pavel's argument also makes good sense that it can be enabled later if self-destruct becomes some something useful but if we uh disable it straight away in shanghai

that's kind of a good uh notch for for that developers not to use self-destruct and maybe not to use cold codes so maybe it's good to disable them straight away yeah i'm still kind of not sure

Tim Beiko: yeah we can definitely discuss that in more detail but um yeah i guess both uh yeah both both uh in person last week in person next week and um on the discord any final comments on eof okay um i guess next thing just like

really quick on the agenda so uh on the last call there were a bunch of like new cfi eips and discussions about that and uh we we ran out of time but i i then asked on the discord to see if you wanted to pause kind of considering new eips for shanghai until we're much farther along with the current merge work uh because we've already kind of made a lot of them cfi and all of the client teams seemed in favor of that um so you know i think it we can still kind of discuss things but uh it makes sense to just not kind of commit to anything else uh until we're much farther along in the implementation process anyone have comments or just thoughts on

that okay that said uh yeah uh enzgar wanted to just take a few minutes to chat about uh and yeah we we mentioned that uh we discussed a while back so eip 4396 which changes how the base fee works uh after the merge um yeah i think he just had some questions about how if and how we should approach this um so and

scar yeah

Ansgar Dietrichs: hi um so basically uh um the situation with with erp is just that um i think it was originally

proposed to possibly be combined with the merge itself um because it's more like a security fix than new feature um then it was decided that was not important enough to be bundled with the merge and there hasn't really been work on it since uh there are still several different alternative versions so there would still need to be some work put in to to figure out what exactly what exact version to to move forward with um i just wanted to basically get a little bit of a temperature check um if this is something that people generally still think uh is important um so i don't know just in case um and you don't remember the details it's basically just about adjusting the way the base fee works whenever there is a missed slot so that um

basically for one you don't have this incentive anymore to dos individual um proposals to basically um decrease the throughput of the network and then also um as a separate second thing to basically um keep a constant throughput even if some portion of the network drops offline in case there's like some issue with a client or something um so basically yeah i just i i was just curious and now that people are like that we are closer to the merch more people that kind of work more with with the merchandise architecture is just something where people feel like this is important and really should should be um like more work should be put towards this or is it something that's more like we can we can put it on like keep it on

hold just wait and see after the merge maybe if there are any real world issues that they would need something like require something like this to be brought to mainnet i don't know it's just for example like the the last shadow folk we did i think there was like um right there there were some some portion of the clients that weren't in initially able to go to purpose box so so it actually propelled behind um mainnet and uh so there's just like one of these example of things that basically this eip would address so i i don't know just just wondering if any clients have any thoughts around that

Martin Holst Swende: um yeah so uh i just i just wanted to get clarified so you so as

could you clarify are there two main security issues that you see with this one being that there is an incentive to dos other datas or block composers and two um the second particular issues would be that due to or events the throughput of the chain goes down is that kind of the essence of the security issues that you see or did i miss it

Ansgar Dietrichs: right right so i would say the first one is more for security concern but just because it actually gives an incentive for malicious actors to do emergency things although again the the inside would only be that you can you can try and bring the throughput of the ethereum

network down and temporarily before we then go right raise the gas limit so it's not like a big incentive um but people have brought that up occasionally and then the other one i would say is less of a security concern and more of just like a network performance degradation concern where like if we have any consensus issues and the portion of network drops offline usually we would have to difficulty adjustment to basically get us back to normal throughput and now after the merge we don't have an automatic mechanism like that anymore so you would stay at reduce throughput until we manually change the gas limit yeah but again that's that's less of a security concern i would say

Martin Holst Swende: and for the second

part there canada could it not actually be a positive thing i'm thinking if we if some nodes go down because it's a very resource intensive transaction okay those transactions are happening uh with 40 percent of all the nodes cannot keep up then lowering the throughput in that case would actually be beneficial for the network um [Music] right i'm not saying like it's always like that but it could be

Ansgar Dietrichs: yeah yeah absolutely so i i i'm just i personally am somewhat uncomfortable kind of relying on these just basically

accidents to like like yes i would feel much more uh comfortable with designing a mechanism specifically around this if people feel like this is a good idea because that would actually force us to commit on it being a good idea i think oftentimes it's more like yeah couldn't it also have positive aspects but um we could of course have something right the way we just look at the last slots and if there's a certain percentage threshold kind of that we kind of go be below or something that the gas limit is automatically decreased or something or something like that we could have have a mechanism like that but i i just i'm somewhat uncomfortable basically just having it be implicitly baked in the network because as you were saying yourself like in a lot of circumstances if there's a sync issue and you just if you're kicked off the

network it doesn't help um to reduce the gas limit so it's i think in in normal circumstances it's more open ish performance issue than something helpful but um again if we usually make a choice about it that's fine it's just something i think we should not just yeah

um because this

Carl Beekhuizen: to add to that i think it's uh it gets worse for the the cl uh if if if you're unable to follow the cl then and and many people are doing this then we're probably failing to aggregate and

all sorts of things which means we're starting to blow up on a memory footprint and that kind of thing so i think it's harder to follow on the cl it helps on the el

Martin Holst Swende: sorry carl so and what's the conclusion what you were saying is that is that you are in favor of the

Carl Beekhuizen: no i i i i i just just to comment on the simplicity and some uh assumption about it allowing us to sort of catch up

as i'm just trying to argue that it only allows us to catch up if we're falling behind on the uh the el but if we're falling behind on the cl then it's probably like this sort of a cascading effect where it probably gets worse

Ansgar Dietrichs: it doesn't get worse because the um execution airbox are less fault right so so when you say it gets worse like the erp will not help that either right

Carl Beekhuizen: no no the the eip doesn't affect that agreed

Ansgar Dietrichs: right so basically um not having the ap could be helpful in a relatively narrow subset of el issues and only if the el

issues happen to be in a range where that exactly the kind of the small amount of throughput loss that you get on the network ends up um being just enough uh to to get you back to sync which seems unlikely that this would ever happen but of course yeah you can you can construct a scenario in which not hanging the fp would actually help the network

i would say i remember there was a stub in the eip about thinking about incentives for changing timestamps um was more thought put into that

Ansgar Dietrichs: yeah that's that ended up turned out to only be an issue

and that proof of work because in the very early days of the erp there was an idea of even bringing this to the domain it before before the merge but after the merge timestamps are fixed so there's no control by the block proposers about the timestamps anymore so this is not an issue but yeah so just just basically say like i think my default here would be to um maybe only work a little bit on it to bring it to a point where we have it in our back pocket in case we do the merchants that some issues kind of occurred that where it becomes obvious that something like this would be helpful um and not actively push for it for shanghai if basically there are people strongly feeling

strongly in either way like either like we should really bring this to shanghai no matter what or like this will probably never be useful anyway then please do let me know but for now this would be basically my default

Martin Holst Swende: yeah so sorry i i dropped off because i fell off the network um so i missed a little bit but i would kind of agree with you and say that i'm currently not yet not really convinced that there's security concerns needs this to go into shanghai but on the other hand i think maybe for correctness we should do it at some point and that's yeah kind of what you said i think

Ansgar Dietrichs: yeah and it's definitely also that's why

i never even brought that up because i don't think that would warrant like really trying to push for it but then it's definitely like usability improvement because it stabilizes the base fee so we have less wobbly of the base fee um but that's that's not an important aspect so it's a nice to have at some point

Tim Beiko: any other thoughts or comments on the eip okay well we might be finishing yuri for the first time in several months um so uh yeah anything

anything more anybody wanted to discuss

Greg Colvin: i just had a quick question uh for martin and alex i think uh i for some reason just noticed that the size of the init code is twice the max code size that is the unit codes at 49 152 so it fits in 16 bits as an absolute jump value but the relative jump proposal is using 16 bits i think so that it can job positive or negative to cover the max code size of a contract

so i'm not i'm not sure how that's going to work silence

Alex Beregszaszi: yeah not sure i was able to to follow but in terms of like unit code the likely use case is that the the trailing bytes are data containing the return byte code um so you don't really need to jump that far in uh yeah the generic init code in this case

Martin Holst Swende: and it kind of it's it doesn't

you can always do change jumps if you want to i mean it's inconvenient but it doesn't mean it can't be done

Greg Colvin: um okay it seems this would be easier with eof so we can separate the code from the data and then it's it's fairly clear that you're copying copying a piece of code that happens to be data but we can discuss that later thanks

Tim Beiko: of course anyone else okay well thank you everyone for joining

um and yes see a lot of you next week in amsterdam see ya have a nice weekend

goodbye so [Music] [Music]

[Music] you

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.