All Core Devs - Execution #159

Transcript

foreign [Music]

Tim Beiko: okay we are live uh

acde159 I posted the agenda in the chat uh obviously to chat about is Shanghai and then some Cancun topics um and then uh payload ID conversation um that's already kind of happened quite a lot in the agenda itself um yeah and anything else that comes up but I guess to start um yeah great work everyone I'm chappella um it worked the network transitioned successfully um there were a couple uh issues like minor issues that we saw we can we can talk about them but I think overall um yeah this was very smooth and

um yeah great work great work everyone um does anyone want to like give or recap um Danny I see you have three comments in a row in the chat so share your perspective

Danny Ryan: those were troll cons um yeah I mean it went really well I think one of the most interesting things that I observed is a number of validators were attesting but not proposing and that's always kind of a an interesting thing because it means their nodes are up uh and they're following the chain and they're kind of boating on things but then this other uh component the block proposal is borked um sometimes that can be an issue with

your local El but given that we have the map boost dependency that's kind of uh also one of the the potential targets there and the person of guys can talk a bit more about that but it was a um a math boost issue one that I think we would hope to see covered in client tests as well as in Hive test so we're kind of circling back to figure out that additionally Lighthouse had some CPU High CPU usage and that was resolved with a hotfix I believe not yet on their staple Branch um other than that I I believe and maybe somebody can tell me if this is exactly what happened we had a number a higher number of missed

slots right at the fork we saw something like 40 000 BLS change operations broadcast because those weren't broadcast prior to the fork boundary and saw High CPU loads so the expect my current interpretation of the highest Miss slot uh boundary right there is the uh just essentially the gossip uh with with honest messages dossing some of the nodes around that boundary so those are the the main points I saw while I was paying attention last night um anyone else

Terence Tsao: I can add some color to the prism issue so this is so this was a prison book that we missed to include the POs to

exact changes when we submit the blind block to the relayer meaning that the relayer will fail the signature verification because of this issue and um so this issue happened because of we uh we miss a unique test case unfortunately also we did not um have we probably did tested this in the um production by production I mean test net but given that there was so few um prison validator that's running with Builder I think probably one to two percent we we we completely missed that so um we're working on post-mortem right

now so that is to be shared by the end of the week and what went well is that the circuit breaker actually ended up saving us because of prison configured to be matched missing uh five slots per Epoch so if we see more than five stars per year part that's missed we will Default about Computing that ended up saving us I believe it wasn't for that we'll probably see a lot more of myth block over e-pod and yeah thank you for um Chris and all the um um or I'll come and come together and helping Lives Many so they ended up um Banning prison validators by use by forcing them to use local for you for you to use local Builders it wasn't for that we probably would also

see more this uh block along the way so yeah we're working on postmortem and then we will um reduce that

further

Tim Beiko: um just one question I just make sure I understand so when you say they're working on they banned prison validators are you saying the flashbots relay basically refused to accept like prison validators connecting to it is that

is that

Terence Tsao: right right right so they so they're able to see through the user agent stream like who is requesting um the header to sign and then you could see there was a prison validator the relay will reject um that validator's request and I believe all the relayers ended up doing this except for one of the

relay area

yeah uh and Chris as you have your hands up I don't know if you want to cut around this yep

Chris Hager: a little bit more color we have been watching the fork live with um the ultrasound guys and with people from prison and from Lighthouse and we started investigating the issues together collecting the data and once it became clear that it's a a signature error by prison we could Implement a very simple check on the user agent that they really does not return any beats for that user agent anymore and luckily we were able to identify prison specifically by the user agent because they do send a particular user agent already and we rolled out this patch and answer to the other relays that were

basically on standby as well also shout out here to Mike from ultrasound we have been in this together and uh basically all the relays rolled out this patch on around 1am UTC let me send the patch here by the way it's a very simple patch notably we couldn't get hold of any of the agnostic hypnosis really people like it was impossible we tried like a ton of ways including calling them like they only disabled these at 7 AM UTC so there were additional mid slots afterwards by proposal connected to technology sphere

in total I have some numbers the flashbots really saw this invalid signature 43 times so that would be 43 Miss slots through the flashbots relay through the hypnosis really 57 times the signature and B slots and the ultrasound relay counted this 21 times so this is all together 121 footy free big release um all the other realistic everyone standby and I applied the patch immediately so I think the overall they were like a two hour window roughly two and a half hour window where the prison request would fail on the capillary call and make their proposes Misty Mr slot

Tim Beiko: got it uh thanks for sharing um I don't know who would be the right person to write the postmortem on that side because that's going to be part of the prism one or uh like any people just I think it's it'd be interesting to capture this information somewhere more permanent than like the call

Terence Tsao: yeah we are working on postmortem right now I think that should cover everything it's also worth mentioning we issue a release um yesterday that's 4.0.2 so if you want to run that boost with your validator then you should be running the release

Tim Beiko: got it

Chris Hager: also like total to run this out like either we can contribute our our data and timestamps to the prison postmodern or we do a another postmortem and drop this somewhere else I yeah both seems good adjuster finally any proposal that uses the new prism release uh this will use a new user agent and this will automatically work again with the builder Network

Tim Beiko: awesome thanks um yeah we can chat online about where it at the info but yeah it should end up somewhere and anyone else have just

thoughts or comments about the prism slash med boost uh situation specifically

Mario Vega: yeah I can add for the hive test here so basically um the problem with Hive was that it's using a glass relayer so this customer layer was the only purpose is just to generate invalid payloads and check that they are well caught by the by the consensus client but the problem was that for the happy happy path it was not uh verifying the signatures so that's why it was missed um I think this can be easily easily fixed in an upcoming uh new test for for hike and we

can catch this issue in

the future definitely yes nice thanks

Parithosh Jayanthi: just a little more color we also probably should have caught it on a shadow fork or one of the other test Nets but at least with the Last Shadow Fork because we had the official Builders and relays on them but we were facing a different issue with signature verification and by the time that issue was patched the queue was empty so we wouldn't have triggered this issue anymore so yeah I guess in the future we also have to try a few more scenarios with the with the relays on

there

Tim Beiko: right then it's like if we can have it relays in earlier earlier Shadow Forks then maybe

um we we have more Shadow Force where they're there and we we end up hitting this bug on one of them

Parithosh Jayanthi: yeah exactly

just about

Chris Hager: not here like we did have the shadow forks and this particular circumstance was just not uh occurring

Tim Beiko: right

Chris Hager: I think on the miniature the fork so maybe there is a additional like steps or protocols you can add to increase like the test coverage on Shadow Forks to catch these types of Educators edge cases

yeah

Tim Beiko: yeah and I think they're probably custom to each upgrade like the withdrawal or like the VLS change Q will probably never be as big as it was on

you know the first block of this work but then thinking about what will be unique um and like kind of stressing the network um when we have the next work and trying to like run through that specific scenario um over and over is is really important

Danny Ryan: real quick pot as you said in the chat that uh you think that the high missed proposals even on that fork boundary was just the the prison issue and not the uh not the bus do you have evidence of that

Potuz: oh no but uh it's just a matter of Statistics right we I mean most of the network about like 80 percent are

sending their blood map boost and apparently according to block print uh 40 of the of those are prism and every prison node would be failing not before if they send to my books so and that more or less matches with the number of missed blocks per ad books that we had after the after the work so I think it's a it's safe to say that most of the Miss blocks were due to this bug

yeah and I might be Fooled by Randomness just when I was looking at the um the the heat map right at the boundary it was like a very high density um and then was less dense after um but that's oh yeah that's because it triggered the it triggered the uh the

failed save right so it started failing quickly and then after that the proposals would start falling back to local execution also the the three layers reacted very fast they banned us very quickly I think it took less than an hour to be already banned you got it but

Potuz: but I think one lesson that we all should be learning is that uh at least I myself when I'm coding I oftentimes think on the happy Faith the happy path which is a local execution and even our unit tests are tested against local execution and we only have sort of like end-to-end tests where where we test

with the Builder and we deploy to test Nets or to to or or even production with the builder but but I think we should set up a a way in that all of our coding is oriented towards testing the Builder because most of our blocks are going through the Builder this is a this is a ridiculous bug that should not have happened at all like there's many places where they should have been tested and all of them failed at the same time this is just impressive on girly for example we had we went back and checked the day of the fork and we only had like three missed proposals out of all of the those that were due to this bug and just they were just lost in random noise which this is something that I should

have tested and we should have tested on a unit test and we failed to do it I guess it's just because we're set up to thinking not on the Builder but thinking on the on the happy local execution path

we

Terence Tsao: also don't run enough um validated through my Boost for our test net we only run like 10 so which means that it's about one percent of the total share of the network so it was hard to catch this

Potuz: yeah we should definitely have a have a network a test net which is more or less stable so that we can actually test our production in which uh at least most blocks go through the Builder

so that that we see this kind of things we could not have seen this on Gurley where we only had like one percent of the network instead of the 40 and everyone sending through math posts as in

maintenance

Tim Beiko: thanks um any other comments or thoughts on the prism bug specifically excited okay um I saw I believe Lighthouse also put out a new release and I don't know if you want to quickly walk through if there's anyone from Lighthouse actually who can walk through um what happened there

if not oh okay so police says this was also on prisms on

prison

Potuz: okay problem so Lighthouse has some CPU issues in the way they process exits and uh and they they failed to cash it correct to Cache correctly the exits and process them correctly when they were missed slots so so they had this bug but it wouldn't have been apparent if we didn't have the missing blogs that were because of

prison mission

Tim Beiko: any other clients then uh have issues or

interesting things they saw during the fork they want to share

just a quick one on thank you if I may um we are seeing occasional spells of slow block import for techu specifically so normally it takes about 100 milliseconds and most of the time it's still taking 100 milliseconds and it's fine but for several minutes at a time this seems to shoot up by a factor of 10 or so to over a second which can sometimes delay block processing in total if it's a if it's a late received block Beyond four seconds making attestations inaccurate uh we are investigating haven't found the root cause yet and it seems to be getting a

lot better um over time it was seemed worse in the few hours post the upgrade uh one hypothesis is it might be due to Patches where there are long withdrawal sweeps you know where most validators have 0x00 credentials but uh we we haven't held it down for sure yet but for uh anyone running techu just be aware of that and we are investigating

Tim Beiko: any other clients anything to share um I guess anything else on chappella

that last chance to bring it up as we move on okay well once again a great work everyone like I think you know um overall this went really well and the network is stable and um withdrawals are happening so yeah very cool to see this finally live

and I guess on to the next thing uh Cancun uh yeah Alex you had posted an update to eip4788 um do you want to chat about basically the changes you've made um and then uh you also mentioned you wanted to have this considered for CFI

so maybe uh we can chat about that and just uh see a fine stuff in general afterwards

Alex Stokes: yeah that works for me so uh this EIP was written a little while ago actually for withdrawals and we went down a different path and didn't really need it and uh that being said I've gone back and looked at this again I made a number of updates uh I'll just run through them kind of at a high level um the biggest thing is moving well even just taking a step back so 4788 the idea is to get some sort of uh cryptographic accumulator from the consensus layer into the execution layer so uh for

example this could be like the state route and that's how the eaq was originally written so committed to the state of the beacon chain and then the reason this is cool is because then you know there's some maybe a mop code or something and you get access to this route and then you can make proofs so you can verify proofs um I guess it's true so that's the drill idea is to have some way to access tickets estate in a trust minimized fashion and the way this EIP originally worked was using the state route from the beacon chain I changed it to use uh each beacon block route instead of the state route the reason why is because the way the beacon chain works is every state has a unique state route but not uh well there can be the same

block route if there's a Miss slot and let me rephrase that so basically if there isn't a slot and that's really the concern here if there isn't a slot then uh the block routes don't change because that's why it's missed there's no block um but the state does change and so what it means is if there is a row of the thoughts then there would essentially with State routes be this like linear amount of work whereas now is like constant time or constant amount of work so we moved to block roots uh the original EAP key to by block number like El block number and I've now changed that to key it by slot uh I think there's a comment from the original uh EIP PR uh metallic went into reasonings of why

we'd want to use slots which uh I think I agree with at this point so you know in short it's better ux um we keep I slot we also so I change this to use a ring buffer rather than just like a you know writing them all to Stage forever just so there's a constant amount of storage that we use um this also mirrors the construction on the CL so that's pretty cool just to have the uniformity there and the way it's written now is that this block route is sent over kind of just like withdrawals are today um and you would you'd essentially just append it uh to the block header so what that means is that it's up to the El to communicate which slots these are for if we keep a slot and yeah does anyone have any questions at

like a high level right now

Danny Ryan: I do okay Danny has the same race oh I just had one clarifying comment which is I think one of the important things to consider here um so by moving uh not by moving to the block route we don't have the linear potential load on the header if there's Miss slot but there still is a minor linear component which you do an s store permis slot so that's just when reading the EIP and thinking about security implications that there is a linear piece right there everything else is

awesome

Tim Beiko: because and I think this is like similar to my question in the chat and the reason for this is because during a Miss

slot you're still gonna store a block route you're just going to store the the previous one like the latest one you have right

Danny Ryan: correct correct so you essentially play ketchup or the Miss slots um there is an alternative scheme here you could actually move the complexity to read for example if you tried to read slot in and maybe it was a flag value like zero you could walk back to n minus one n minus two Etc and like have a higher complexity on read and have and once you hit a non-zero value like return that um and have a higher potential cost gas cost there um given what like the overhead of of the block processing

and the right here and the potential of Miss slots and that kind of stuff it's I think much preferable to put the complexity in the right but that's an option

Alex Stokes: right so uh well yeah I guess so the thing is um there's one issue I see um yeah maybe just quickly if anyone has any other questions so far okay so I'll take that as a no and then that is written so yeah everything Danny said is correct uh that's an important distinction and also right so the one

thing is that as written this is assuming that the eels in essentially use the time stamps from the headers to drive slots which I don't think they have the functionality now I don't think it's being added say from 4844 or like anything else that might go into Cancun so um yeah that's a question for everyone here do we you know are we comfortable adding this logic or would we rather not sort of leak that uh abstraction barrier between el and cl

Danny Ryan: and what is the alternative to read by

well you

Alex Stokes: could just you could just send the slot over so you know you're already trusting the CL for the roots you basically also just look for the slot so

that just means there's more in the header and you could imagine like having the start slot to end slot you could like write the start slot to the state so that you only need the one slot number you can send them both there's a couple different options

right

Danny Ryan: or you could like try to have this buy time stamp and have bad timestamps be failure cases on read and like other weird stuff but that would bring a lot of moderate amount of complexity into the construction ring out of degraded

ux

Alex Stokes: right so for the El to compute these slot numbers which is also something apps will probably want at some point uh

you need two pieces of configuration from the CL um so it's not too much but you know it is something and I think the bigger thing is that it violates again the subtraction we have or like the the barrier between each layer

I'm a cow your hands up

Mikhail Kalinin: yeah I just wanted to um to add that we we still have a nonsfield of el POG header unused which sounds like a good place for for a slot so it doesn't add any data complexity and potentially we can expose slot on the evm level in some in some future if you think that it will be useful so um

and avoiding the com derive and Slot from timestamp on the hillside is a good thing in my opinion because if we at some point in time change it um I mean like the slot duration so it will have to it will have to become the configurable parameter so more more configs more time between the two layers if it can be avoided that would

bring and under oh sorry Alex are you gonna

say something well I was just

Alex Stokes: going to call it El devs because I see many of them here and I feel like don't have some opinion what are you going to say Tim

Tim Beiko: I was gonna say is there so so if I understand mikal correctly it's like you remove the need for just pre-compile that you're writing to um you instead have the slot number as part of the block number for free as you know the non's value

right

Alex Stokes: so yeah so yeah so

well there's kind of two things so one of them was just exposing the slot to the evm as well and that's like a separate Downstream question but then Upstream is like yeah how do you even get this data to the the El in the first place and one option is rather than try to look at the timestamps and the headers and compute the slot you would just pass it along with this blocker as well and the question then is where does it go uh there's apparently this knots which I believe is u64 that's the right size it's just sitting there and used so we could put it there

Tim Beiko: and yeah and I assume there's no non-op code right like I don't think this was previously exposed

Alex Stokes: don't think so yeah

yeah no

Tim Beiko: I assume we would have dealt with it during the merge if that's the case

Alex Stokes: hopefully

yeah I'm curious if any yell dabs have opinions thoughts questions on this and

Danny Ryan: we can give some room to talk about it in one week on AC DC uh obviously from a consensual perspective but if El's come there we can also hammer it from that angle as well if you have some time to look at it

yeah so I'm

Potuz: I'm a bit confused about this difference but so are we talking

about like exposing the slot number on the evm uh on every blog instead of just making the accommodation from Genesis time plus this number of uh of on this timestamp just because we might change the duration it seems to me that the complexity of changing the slab duration is going to be just adding a constant after a particular pork and which shouldn't happen in many forms that we're going to change that slow time to sort of like adding data on the evm on every single block to prevent this extra computation seems crazy to me

Mikhail Kalinin: I'm not sure what you mean by

on the evm it's just you know passing a slot into El header and we have a place there which is uh remains unused since the merge so there is a field a few Institute for data type which is filled with zeros currently

Potuz: we could use something useful to send instead of the data that is completely equivalent to the timestamp that we have already sent in on that header

Mikhail Kalinin: oh yeah yeah I see yeah yeah that's fair but we will have to keep to have this complexity of maintaining solid duration on the outside yeah I got it

Danny Ryan: I have a a quick question so often when I think about the state transcription function usually I want it to be a function of the pre-state the block and then it outputs the post State um and I just realized that I believe this actually makes uh this a function of not only the pre-state the current block but also the previous block because it's doing a read from that rather than reading the time stamp from the previous block from State um is that a concern that's usually kind of a little bit of a red flag to me when we're abusing and extending the inputs of the station

transition

Lightclient: function I

mean we also already need the last 256 blocks for the block hash op code

Danny Ryan: right but it's all that's maybe something we don't want and would want to fix so I wouldn't want to like add another dependency of previous block

Alex Stokes: yeah the reason that the CIP uses the state and the way that it does like client is for that reason like there was desire in the past to move away from having the states yeah the the pre-states block Post State but also there's almost this like hidden input of this history element

Danny Ryan: yeah but there's another hidden input right here which is the previous block header

Alex Stokes: all right so that that can be changed to write

Lightclient: I mean yeah it

depends the previous block header for the 1559 verification right

Danny Ryan: you tell me

Lightclient: yes we do

Danny Ryan: yeah okay so the so the Elsa transition function is pre-block header previous date block currently yeah and likely won't change okay then that's fine I just don't want to extend the dependency

Lightclient: yeah I agree yeah without consideration the 256 because we I think everyone thinks that's

degenerative

Danny Ryan: okay thanks

Tim Beiko: so yeah there's some comments in the chat about exposing it to the evm as well so I think there's like two

separate things right one is like where do we store this data whether it's the states the nons uh something like that um but then obviously how do we expose it uh in the evm unless you maybe want to take like a minute or two to talk about like why we should expose this in the evm and what's like the value that you get out of this

Alex Stokes: well are we talking about the slot numbers themselves or this route

Tim Beiko: either

uh well I assume what do you think is like yeah correct

Alex Stokes: sure I mean it so right I mean there's all sorts of things around

Bridges and my clients and sticking pools and you know all sorts of things I mean like immediate use cases would definitely be around like more trustless sneaking pool designs where like right now a lot of them just like have some Oracle that you know there's like a multi-stick updating or something like um and then instead you could move that to again just like proving some bit of the states anyone can do this and it reduces a lot of governance risk for all these staking pools fighter rocket

pole

Danny Ryan: you know more yeah I think importantly it also reduces the barrier to entry to create something um because getting a good Oracle is hard

Alex Stokes: yeah that's a good point

Lightclient: so was that the argument for putting in the

state or this is the argument for the op code in the first place

all

Alex Stokes: right so we're just talking about and the so well there's a number of decisions here because it could also not be an off code but basically this is just the idea of having the root to be exposed in the evm so like that's what the CIP sets out to do that's the important bit uh I don't think we should get tied up and like is the sun exposed like what does that look like all of that stuff

Tim Beiko: and sorry just to make sure I understand uh why like what's the value that you get from having this slot number if it's not exposed in the evm itself

Alex Stokes: well so it would be written in this contract and then you want to read it so

in that sense it's exposed

Tim Beiko: it's in the state okay yeah so so so just I guess so mikhail's idea so like mikhail's idea of pulling it in the nods if we does if we do that then it's not written in a contract and so oh no it wouldn't be so

exposed and so

Alex Stokes: no no that's so that's just essentially to like commit to it well let's see I mean maybe we could think about something weird but it'll end up written to the stage according to the current EIP

Tim Beiko: okay correct yeah so it's like because then later as a user I go and say hey what was it at this box and I give it the song number yeah yeah yeah okay and then I can imagine also potentially

if we if we are writing it to the state as well and we also write that the block header then like off-chain applications that like rely on block headers can get that information without having to read the state right and yeah let's I assume the extra benefit that you get from having it there

having the roots or some slot

Tim Beiko: no having the slot member sorry having the slot number in the block header instead of the nons

Alex Stokes: well it needs uh well okay those are the same thing to me that's just a question of if it's appended or if it's if it replaces the

nods

Tim Beiko: I'm not sure I understand so you're

saying that sorry the root is the same thing as the slot number because you can

no

Alex Stokes: no so you need to send over you need to send over the root you need to serve the root and the slot number and so the reason you want to put in the header okay so you want the root in there the reason okay you have to have both of them in the header because for sync actually so I was trying to think if there's some way you could like not have the slot in there because if that's breaking abstractions and all this but either way um if I'm also like syncing of work for some reason then I need it in there because the CEO won't be able to tell me um so whatever it needs to go into the into the state will end up in the header

and where we're at now is having the root and then one slot number

Tim Beiko: okay thanks that that makes sense and I saw you like briefly had your hand up I don't know if there's something you wanted to add

I I was just maybe I was trying to clarify basically between the root and the number and everything but I think Alex did a good job

clarifying

Tim Beiko: does anyone else have questions or thoughts on this

Lightclient: I think the only thought I have is I would like to

consider more adding this as a piece of the State versus just having it as a like ephemeral singing clients to fill the op code

Alex Stokes: got it

Tim Beiko: is that not what

yeah

Alex Stokes: do you mean getting rid of it or keeping it

Lightclient: getting rid of it out of the state I'm trying to convince myself that it makes sense as a piece of the state

Danny Ryan: yeah I mean the main thing is uh if you only have it as a like a so if you don't have an estate you can only read the

previous header right because you're gonna the previous speaking group because it's the only thing that's going to be available because it's going to be in the header um and at that point you make the ux of sending transactions um really tough like you have to your transaction has to be for precisely the um or it has to get into the next block because otherwise you're going to make making proofs against the root that's broken with respect to your proof so the state and then being able to bound your proofs against a particular slot uh make it so that you don't have like weird failed transactions and bad ux around using the top code is that what you were considering

uh I

Lightclient: my understanding is that you have access to 8192 roots is that wrong

I don't know

Alex Stokes: so yeah what Danny was saying you could do that still with what uh like clients trying to do with having uh this like extra history parameter so this point is just like do we want to have this like implicit history or not uh you know vitalik and others in the past have definitely wanted to be away from that I think it's cleaner Danny put some comments in the chat around how we handle this in the CL and again I do think it's cleaner

oh claiming that you should just have them ephemerally hanging out with a client yep oh

Danny Ryan: um that's really bad for statelessness right

Lightclient: I mean why like we already have it for Block cash and you have to have this

data whether it's part of the state try or not it doesn't seem right

or

Danny Ryan: if it's if it's part of the state try then it just becomes if people are accessing it um in statelessness it becomes part of the proofs whereas if not then your client in statelessness has to have this data locally and like have gotten it from previous blocks or somewhere else and and and the proofs so it makes it just kind of adds complexity there whereas obviously we have that complexity with the 256 block hashes but reducing that actually makes like I would want to eliminate these things so stateless is is cleaner um it has fewer dependencies rather than

add to them

Lightclient: yeah okay I think that makes sense

Tim Beiko: I guess given um the like amount of back and forth on the design on this um it probably makes sense for us to discuss it like next week on the CL call once people have had a bit more time to like wrap their heads around it this yeah does that seem reasonable no no breakouts I don't think this this war it's a

breakout

Danny Ryan: we can do this one I think yeah and we have a low agenda next week

so we can spend 30 minutes on it it's fine uh oh sorry about

yeah oh

Lightclient: I should say obviously I think that this EAP is pretty important and that we need to come up with a way of getting the beacon block route into the evm I just want to come up with the best design

Tim Beiko: and I guess yeah in terms of like the CFI conversation I feel like now that uh Shanghai is completely out of the way does it make sense to have the next cell core devs uh like yell call focused on just generally what are the things we might want to see

if I for um for Cancun um so that we're not like just making this one like one-off decision now but like we can also have like two weeks for teams to just look at the set of things that are being proposed and try to think about like what's like a coherent set of things we want to do um and even if we don't agree to like everything we want to do in the next week like maybe we can like flesh out the one or two most important things on the next call alongside 444 yeah just I don't know so kind of we don't just like add things CFI one by one and then realize we have like five VIPs we've committed to

um without having like a higher level View anyone strongly wants to like CFI this now we can have that conversation as well I think otherwise having like a higher level conversation in two weeks probably makes sense last call for objections okay yeah so let's do that um and I guess over the next two weeks as well we have this eighth magicians thread um I believe all of the eips that have been proposed are either Link in that thread or there's also a tag Cancun candidate on its magicians um so I'll make sure to update uh the

main like the first post on that thread with everything that I'm aware about about that has been um proposed so people can look at that if there's anything missing there feel free to ping me um either on Discord or Twitter each magicians and so I'll make sure to keep it up to date before the next call um and then yeah if client teams and and just folks generally want to like think through the various options um what we should prioritize I think that would be uh valuable yeah does that make sense

next up then uh I believe it was Perry you put this on the agenda uh the idea

of starting devnets with capella Genesis uh do you want to take care of Mr chat about that

Parithosh Jayanthi: sure uh yeah I just wanted to bring it up to see if any client teams have already tested support and if not if we could start testing support so that we can start testnets from capella instead of um politics Genesis States and also asking that in the context of if that's supported in all the feature branches for the different um features so I know for example we can't start from capella Genesis on worker yet but if it does work on eof as well as on 4844

already

Gajinder Singh: for load star it should be possible to start from capella Genesis

any reference

Parithosh Jayanthi: cool I can also reach out to the client teams separately later on thanks test net that's currently live that was meant to be public and test withdrawals we'd like to officially announce its application and we hope to take it down next week Wednesday

so if anyone's using it for something active please let us know um otherwise it will be deprecated by

that

Tim Beiko: um and I have four calls like two weeks ago we were talking about setting up a longer lived 444 devnet in the next few weeks um or so so um I suspect yeah once we have that up we can point people in in that direction as like the main thing to go if you want to test The Cutting Edge features before they go on public that's not then okay anything else on Cancun

Danny Ryan: uh just real quick the way that we do

releases on the consensus specs is their release candidates until the rules are actually on mainnet the rules and logic is now on mainnet so in the next couple days we'll be releasing a um consensus specs non-rc release this will also include the transaction type change to 0x3 on 444 so I believe that will kind of give us a good Target for

our foreign

Tim Beiko: nice and this reminds me uh we should Mark all the eips for Shanghai as final now that they're on Main met

um so VIP authors uh of the Shanghai eips um yeah if you can open a PR just to move it to final given that they've all been activated as well sweet anything else on Shanghai or capella

Mikhail Kalinin: okay if not Mikhail you had posted uh the engine API PR discussion uh there's a bunch of people who chimed in on the agenda already but uh do you want to give some quick context on that then I can discuss it okay so um the original problem is that in

Nimbus and basil pair combination there was a updated with payloads at payload attributes initiating a build process build process so let it resend this uh in the same slot if I'm not mistaken um the new pillow that attributes with the changed with goals and the problem is that bezo did not Factor the factor in goals into the um and the payload ID computation process so they did not restart the payload process and yeah

that led to um failure proposal and it appeared that we didn't have anything about about that in this pack saying that the payload additions unique they identify the build process and be anyhow related to payload attributes and yeah hence the reports fixed to the spec um if I'm yes if I understand correctly there might be some other yellow clients that are affected um and probably guys somebody want to chime in and share more information on that but with respect to the fix so the fix

is here and there is a discussion happening in there um so on to engage El Clan devs to to look at this VR and if it is okay um say something or participate in discussion the idea also is that by having this in this pack having the statements in this pack we can yeah we can then create the um ACT test and then Force this Behavior to avoid such problems in the future that's it

Tim Beiko: thank you um Lucas you have your head up

yeah

Łukasz Rozmej: so unfortunately another mind also

um doesn't didn't doesn't include the withdrawals when calculating the payload ID so we will be fixing that soon

Tim Beiko: um and there was a comment I guess uh by gajinder on um the agenda where he said it should not be possible for cl uh since withdrawals to like send the two payloads that are yeah two for Chase updated messages that have the same attributes for everything except withdrawals so he was curious when um like what would trigger that

I don't know

um this should be bargain seal side um okay I mean like

Potuz: yeah this is probably a bad cachet head State like sending the windows for a different head state

Tim Beiko: got it okay and so this fix would mean that even in the case where you send like this buggy FCU message you still get a different payload ID because at least one of the attributes has changed is that correct

right

Mikhail Kalinin: yeah and the new build process should be started to you know to factor in this new information and withdrawals in this case for instance

Gajinder Singh: yeah but the proposal will still fail because withdrawals shouldn't be any different right so if it's giving different withdrawals then probably it has calculated the withdrawals incorrectly and The Proposal will eventually fail

Tim Beiko: yeah Mikhail

Mikhail Kalinin: yeah yeah yeah just yeah yeah there was a um yeah the the failure is because there was rules were sent

had different values so yeah that's that's that failure so it's basically not because of the Basu only in this case it's because of the bargain CL side and this behavior and on your side as well so which didn't catch up with the new roles information it sort of makes so but yeah I would like to you know to you have this I think it makes sense makes a lot of sense to have this in its back and enforced by hype test as well

Tim Beiko: um I guess in terms of next steps here

should we just go ahead and merge has changed I saw Proto had a recent comment that's all I didn't have time to read um but should or should we maybe leave it open a couple more days just to make sure we address all the issues or like the comments on it then yeah close it then

Mikhail Kalinin: yeah yeah I would I would leave it open for the next several days probably for a weekend yep then just merge

Tim Beiko: yeah and at the very latest then on next week see alcohol we can agree to merge it

but probably oh yeah yeah just

Tim Beiko: do it before yeah okay sweet and

anything else anyone wanted to cover okay and Meryl's saying in the comments we can add a hive test for uh for this case as well okay otherwise maybe like weekend yeah

Mikhail Kalinin: yeah we can basically work on high tests in parallel so it's it still makes sense

Alex Stokes: if there's anything else I have one thing to bring up around Med boost

Tim Beiko: yes please

Alex Stokes: so uh there's an RPC okay so this is about verifying blocks from Builders as they go to the relays and there's a custom RPC that flashpoints implemented in their relay uh and right now basically it's just a fork of death this came up on the med boost Community call uh uh one or two calls ago where we'd like to have client diversity uh at this layer of the stack as well and what that means here is having other yells uh El clients implement the same RPC um something like basically having a standard for this and then everyone's supporting it is this something that people are interested in doing do they

see the value in this do they think it's a bad idea um I think probably if we all think it's a good idea then I can go and like work on some sort of RPP RPC spec but I just wanted to get some sort of initial temperature check before I go and do in point is saying given a given a payload like is this payload

ballot

Tim Beiko: and in practice I assumed we would have this in the execution apis repos does anyone think like this does not belong there

Danny Ryan: okay wait would this go into execution

or Beacon apis

Alex Stokes: so there's a separate uh discussion point for having this essentially having the the CL Parts be at the CL and this is essentially the ale part happen at the El okay good okay you kind of need both in different places um but yeah this is just for having this be a standard API across eels so that because it's very not basically all the relays are using the net fork and cuff and given how many blocks come through the system you know it kind of harms all the efforts we've done with client diversity at that part of the stack so it'd be nice to have optionality here I'll take the silence as compliance

Tim Beiko: yeah

yeah yeah okay yeah so will I so I think yeah you can just open a PRN execution apis repo which is where we have all this stuff um specified now oh uh yeah

Miguel

Mikhail Kalinin: um I'm just recalling that we were agreed to double check on this thing which is Builder override flag before I get payload um so what we have decided on the previous call is that if any El team is willing to implement these heuristics for this this type of black before the

next cardboard then it will make sense to you know to stack it out right now and start working on it um but as far to run no like attention to this PR so I'm just you know um reading this as there are no um intention to to work on that so I think we should just push it to to the um do the next hard work and include it in the scope of or engine API changes that are scope so if anyone thinks the opposite so that's just yeah go to the pr in the comment

Tim Beiko: yeah sounds good and there was a comment on the pr um I think by Alex who said it would be nice to see some prototypes of this working uh before we decided to add it the engine API so I don't

Potuz: know on the CL side we can do this immediately uh and on the El side I think uh Marius has a draft I don't know how much can that be

worth

Alex Stokes: and just add some context the comment there was essentially saying like this could all be done outside of having any formal stack or API that we all agree to

and it'd be nice just to like prove that out right so like you know if the concern is like some sort of censorship resistance detection then actually prototyping that out um ideally and like you know clients on midnight but uh yeah just hop in some use case Beyond like oh yeah there's this like additional thing that everyone has to deal with for all time moving forward and like it'll probably do something without actually you know doing

Potuz: so in my original issue I added three different heuristics that I think are pretty valid uh and they are they just have a couple of constants that are

configurable so by product prototyping them you mean having some email that actually implements some of those heuristics at least right

Alex Stokes: yeah it would be nice I mean again this is not a blocker but oh

because otherwise my concern is otherwise

Potuz: no but I think Misha is actually proposing this as a blocker which I agree I think we should have at least some sort of like draft PR of Samuel that actually wants to implement some of these heuristics before we push to have this in the in the specs

Alex Stokes: great yeah

Tim Beiko: anything else

okay then yeah let's wrap up yeah thanks everyone um congrats again on chappella and talk to you all in this CL call next week

thanks thank you thank you see you guys

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.