All Core Devs - Execution #141

Transcript

[Music] [Music]

Tim Beiko: um

are we live i'll assume we are um so good morning everyone uh thanks for coming to all core devs number 141 um a couple things on the agenda today um first just a shout out on grey glacier uh which is happening next week uh then a bunch of merge uh related stuff uh then uh like prime had uh had a small issue around uh the debug name space you wanted to get feedback on uh and then finally uh paul from lifehouse here has has made a suggestion uh for changing the day that

we have these calls on to be more friendly towards um the parts of the world where it ends up being friday night or saturday morning so we can chat about that at the end there are two eips that uh uh we've been bumping from call to call to discuss and never had time for them uh but then uh one of the champions has some issues that they're looking into for eip 5027. um but then for eip4444 there is also like an update that was posted on the github there so we can discuss both of those on the next call but if folks want to check it out async uh in the next two weeks um so at least we have the context when when we're talking about them um first yeah so great glacier uh

happening next week uh it's expected around july uh sorry june 29th um there's a blog post with all the client releases out in them if you haven't upgraded your node now is the time to do it i don't know if there's any client team who wanted to chime in about the upgrade okay um so uh yeah update your notes for that um next up uh moving to the merge stuff uh we had our seventh mainnet shadow fort uh earlier this week um

perry do you want to give a quick overview of just generally how the fork went and then i know that like basu and nethermine had some stuff they are sorry basia and aragon i had some stuff they wanted to go over as well but uh yeah do you want to walk us through high level how the the fork went yep um so we hit uh the chain was launched i think last week on friday or so and we hit ttd around wednesday after an override like a day earlier uh we hit tv a couple hours later which i guess is a sign that hashed is changing even drastically um within like a span of a day

but irrespective we hit ttd and we were finalizing for a while i think we immediately dropped down to around 80-ish percent um so about a 20 drop of our notes and over the span of the next 12 hours a lot more nodes dropped offline uh later we figured out that most of the nodes that dropped out for either aerocon or bezel nodes um as far as i know the the aerocon issue isn't really an issue but rather something related to how shadow folks work um they're looking for a block but essentially don't have a peer who's providing the block to them so they get stuck um there was i i think i had enabled the experimental

overlay and there was an issue with each stats that i had enabled on the conflict so i removed both those flags and since then aragon is progressing i think we saw like two or three percent come back online hopefully more does come back online soon uh the bezel issue however is a deeper one and i'd let someone from the bezel team talk about that

Justin Florentine: yeah i can take that this is uh justin from the basic team if you want me to go now uh just one second this is one more update yeah i've updated the same tracker we have for uh checking for proposing empty stocks um i don't have any data from

uh peso of course but from the other ones the only combination that proposed an empty slot so without transactions would be in nimbus aragon so we're doing pretty much as well or slightly better than what we did with roxton so that would mean there's no real incompatibility right now but there's no regression at least

Tim Beiko: nice um yeah i don't know maybe yeah thanks thanks barry um maybe uh does anyone from aragon want to go over to the purin one uh before if this is like a bit a bit simpler and then we can we can do basu in more detail

Andrew Ashikhmin: yeah i can briefly describe the issue

the issue is not per se with the merge but with the shadow forks is because we have kind of we don't have a network split so we have peers from the shadow peers and um upstream vanilla mainstream ears and uh occasionally we don't have enough peers from the shadow fork so we can not download block bodies or something like that and uh everyone becomes stuck but i i don't want to spend too much time because this shadow fork testing is a bit artificial it shouldn't happen uh on on real test nets or on real

mainnet would it be so we also just did a small flat update uh we just added static peers and that pretty much solved the issue right then like for future shadow forks is it like straightforward to just we could literally peer all the shadow forking nodes together right like um that we run yeah i think from next time onwards i just use static wheels that might just be easier right okay um anything else on the aragon side nope

okay um justin do you want to walk us through what happened on

base here sure

Justin Florentine: um okay so with regard to uh mainnet shadow fork seven um we saw a long running issue that the team has been tracking get dramatically exacerbated i've attached a hackmd write up that goes into much more detail to the um agenda in in the comments there feel free to read that and send any comments my or anyone on the team's way but the nature of it is has to do with our bonsai storage system and a concurrency issue that we found in it in the past this has been fairly rare and we've never seen it actually hit all

of the nodes at the same time around the merge like happened on shadow fork 7. we don't think this is merge related we still maintain that this is a concurrency issue in bonsai and we have been working this issue pretty rigorously since it first showed up in late april uh we've made a lot of progress on it and we do have some ways to reproduce it now that we've got better hive test adaption the fallout though was basically made up 25 of the network and all the basic nodes failed so we started to fail finalization participation dropped there wasn't

enough to keep maintaining we did continue to propose blocks but all the blocks were empty so that's not terribly useful um so this is an issue we are tracking it we have made it we have managed to make it worse which we actually like because it makes us easier to reproduce i think you all can appreciate how hard concurrency issues are to reproduce and analyze um we are continued to working at um to work it and um if there's any need for greater detail i'm happy to elaborate and i hope everybody can get to the the hack and be on the on the subject

Tim Beiko: thanks um i have a quick question i know i think bonsai trees is like the default

storage mode now if if that's right but i assume that non-bonsai storage still is supported in basic is that

correct yes it is forest is still supported forest is actually still the default uh bonsai force is still the default

Justin Florentine: yes bonsai is no longer experimental um however it is not the default okay and i assume just because like bonsai is is is smaller from a disk size perspective that's all we've been using for for shadow forks but um correct yeah and so if yeah i guess a quick fix there and this might be hard for like mainnet

but it's like using or actually no because it's the sync but like yeah i guess a quick fix is we can just use some non-bonsai nodes on the basic side and and see if those um yeah if those work smoothly yes we'd like to suggest next time we do a mix of forest and bonsai nodes for the next shadow fork yeah we've also discovered that this is more exacerbated when running in docker um our canaries that we run don't run in docker and we um very very rarely see this issue we do see it it does happen outside of docker but docker does seem to exacerbate it which kind of led some credence to um you know timing and concurrency issues right okay anyone have thoughts questions on

this okay so it's encouraging it seems like the two big issues we hit are like client issues but not merge related issues um so that's that's good that's progress um anything else about the shadow fork in general

Marcin Sobczak: i would like to add that there was 10 nether mic notes and all of them was successful on transition and right now eight are still healthy

but on two we have three exceptions like both notes with deco are down for now and they're investigating what's going on

okay so the nethermine tekku pair did have any issues uh yes but it was like day or even more than day after transition after

Łukasz Rozmej: that i'm working on a somewhat related fix so this might get fixed by this but we are not entirely sure if this is the

the truth cost

Tim Beiko: got it okay um and i know there were a couple other merge like related topics um around like latest valid hash and and sync that we wanted to chat about so i think it probably makes sense to get through those uh before we start talking about like the next shadow forks or sepolia um yeah just to make sure we're all kind of on the same page um but yeah anything else before we use that yeah and i have another update um but this time regarding robston uh so sam from the ef devops team um

unfortunately he's in australia and couldn't make it to the call but he wrote a really nice write-up i've just shared it in chat right now about the automated sync test that we have that he ran on drop screen um all the code and the repo hand chats everything is open source so people can reproduce it if they want essentially he syncs up the head on robson on every combination then he runs complexing tests for example stop the el for one epoch and then restart the er stop the cl for one e probably start the year or do it the other way around or fully sync yell and then start genesis sync on cl and so on um

yeah the results are there and i think some tests uh need to be real on but essentially at least the sink to head tests uh mirrored the results that the each takeoff guys found as well and as far as i know those already have open issues nice um anyone have questions comments on that

Łukasz Rozmej: yeah i have one so are those tests like available to be rerun or being run as somewhat like hive test or something that's because we are still working on that code and you would like to uh it would be great to have some kind of tests to potentially spot regressions exactly uh they are um

let me just link to the repository it's just a github actions thing similar to the nightly run uh you can make a pr over there with whatever you want to change and it should be reflected so we have a sync test coordinator and then it triggers off the sync tests by github action and then the result of the sync test is also viewable on the data action but i thought it's easier to give an overview with the table instead in case you guys have any questions about the setup um please reach out to sam he's on twitter and discord and i can connect you if you need to

Marius van der Wijden: i

i also ran my own robson sync test today and what really stood out was that i ran lighthouse gas and lighthouse took 50 minutes to sink for i think three weeks worth worth of blocks now and gas took like 15 or 10 minutes and so like the the consensus layer sync is way slower then the execution layer sync and it's just something to keep in mind

Paul Hauner: i think the lighthouse sync has some a bit of improvement to do as well we're adding some concurrency and i'm starting to do some benchmarking about how fast we can sync with the merge um i think in general the consensus sync is a lot slower than execution layer sync but um the checkpoint sync is kind of what we need so it's not hasn't been a huge target to improve syncing speeds in the consensus layer

Marius van der Wijden: how how long would you expect the the checkpoint sync to like how long do we need to sync with a checkpoint sync like two weeks or three weeks or how

often would we do the

checkbox

Paul Hauner: we can usually do it since the last finality so i only have to sync you know like 6400 blocks or something like that i think um keku and lighthouse um we'll do a checkpoint sink in less than a minute around about a minute

Péter Szilágyi: yeah but i guess for in order to do a checkpoint sync you need to manually

checkpoint right

Paul Hauner: uh yeah you can also point to like another um beacon node endpoint and it can pull it down so you can use like a centralized provider or you can use um another node that you have checkpoint sync is a requirement for proof-of-stake thinking basically um

forever any any period after genesis of a matter of weeks

but

Péter Szilágyi: uh i guess the idea is that somebody could post a checkpoint every so my expectation with checkpoint thing would be that you pull in that checkpoint and then you just use that for the next three months or five months or six months and then eventually just choose another one but i mean if it takes super long to synchronize even from a two three week or checkpoint and it gets a bit annoying to always have to go somewhere and catch it the most i i can imagine most people will just configure their

notes to just look at infer and do whatever inquirer says

which kind of beats the purpose yeah

Paul Hauner: baking in like a three month six months uh checkpoint state um isn't how it's gone with the consensus clients there's been a bit of talk around it i'm not sure if you want to discuss it now but that's that's not the way it's

gone

Paul Hauner: um yeah maybe this is a good one for discord or something like that

Péter Szilágyi: yeah anyway i think mario just wanted to bring it up because there's this big discrepancy and uh essentially the experience that people will have people who are in ethereum may not know currently is that synchronization could

become even catching up could become super slow and would be nice to at least file too much load on top

from the beacon price

Paul Hauner: yeah i agree it would definitely be better to make it faster i think the um the staking community are pretty familiar and comfortable with checkpoint sync as well

though yeah and only a problem with validators or stickers though right this is a problem for anyone

Paul Hauner: yeah that's right i guess the point i was trying to make there was that people who are used to running beacon nodes or

people who even people who've just started out running beacon node seems to seem to get to speed with checkpoint sync pretty well they're definitely not i'm saying we don't we couldn't um benefit from improving consensus lasting speeds um yeah i don't i don't think we're gonna we're gonna improve the full six spec on the cls right now but um definitely something something people should be aware of um i don't know i guess yeah we can continue the shout and the discord uh either the consensus uh dev channels or aqua devs

um just to take a quick step back any thing else marius you wanted to share from just the sig test you've been doing

Marius van der Wijden: no it was um oh yeah but before uh the merch happened we i think with pretty much exactly 50 slots per second and after the merge hit um i synced with 30 slots per second so um this is the like the engine you payload and forge is updated and so yeah that's the default synth with

the with the blocks it um thanks and andrew i see you've had your hand up for quite a while so

Andrew Ashikhmin: um yeah so about checkpoint i think do we have inferior for robson because to my mind it's very important to check uh to test the checkpoint thing because for for the mainnet it's pretty much a must so we should definitely test it more on robson and other test nets and to my mind it should be the default for for like all public test nets you can opt out but by default checkpoint sync should be

should point to inferior or something that that's my thinking

Paul Hauner: yeah it's always been pretty hairy putting in fury and as a default that's why we've um struggled to put um any default checkpoint syncs in i'm not sure if there's an infuriat or a beacon an api provider for robson my guess is no but i could be wrong

Péter Szilágyi: i mean just right you're just mentioning that for robson for the test it probably makes sense to just point it at inferior and not care because there have been is it should be as pleasant of an experience as possible but for mainnet i think it would be a very bad idea very bad public view to just point out so

yeah and and uh i'm not sure that there is uh whether it's inferra or like another that there is a like api provider for the robson beacon chain obviously like a bunch of these support drops in at the el layer um yeah and i also feel pretty strongly that like whatever decision on that front it that's much more like a client level decision than like protocol level decision uh so you know you could imagine lighthouse using infera and prism using alchemy and like nimbus not using anything you're like using something that they maintain so um

yeah i i feel like this is probably it's not something i don't think we have the the like checkpoints to do but um also if um yeah if if we do it it feels like uh enshrining any single provider uh at like a protocol level spec fields it feels quite wrong but the clients are obviously free to do whatever they want in terms of like improving the ux for their users and yeah

um i don't know if anything else i'm

Paul Hauner: just gonna say it's potentially an interesting point that um we might not we might be under testing checkpoint sync on on these test sets um

from a um i'm mainly interested in it from a um execution layer perspective um just for execution layer devs it'd kind of look like your consensus layer just comes online say like in the last like say 128 slots of the chain and just start sending your payloads from there and then sinks to the head and then just waits for you to catch up um so maybe something to think about is trying to make sure that we test checkpoint sync a bit more with um execution and consensus layer clients i know at lighthouse is not something we've really thought about before maybe someone else has

Péter Szilágyi: so that's that's essentially the preferred way to synchronize a guest note just give us the head as soon as possible and we will figure out the rest

so for for get at least execution-wise the later header you can give us the better we don't care if there are gaps i don't care about beginning of the

change just give me the head and i'm happy with it yeah it's similar for nevermind and for aragon

Péter Szilágyi: so i think that's kind of all execution layers who can uh snap think work saying fasting or whatever in order to do that they need a head header or something as close to

head as possible so that's kind of the expectation

great

Mikhail Kalinin: probably we should just pay more attention more users attention to using the checkpoint sync in the documentation or

any other places i think the idea isn't necessary that it's hard to use a checkpoint

Péter Szilágyi: or that users don't want to use it i think it's more like they have no idea how to use it or what to point it at so if we could somehow make that obvious or some helpful to probably go

a long way

can the

Łukasz Rozmej: checkpoint be like download through peer-to-peer like why is it like you have to point it somewhere i don't understand

the design the point of checkpointing is that you provide a hash a trust attach or trusted checkpoint

Paul Hauner: yeah so how it works in um consensus clients now is that they go and query a beacon node api so the standard http api on beacon nodes and then they get um i think most of them will get like a re get the finalized checkpoint and then get a state download a full beacon state which is order of 100 meg and then a beacon block uh and then they'll start syncing forward from that the reason

the idea behind it is it's supposed to be like a from a source that you trust so like your friend or something like that um how that turns out in reality is slightly different but um the reason that we haven't done it over the peer-to-peer network is for two reasons one because if you get eclipsed at the start um like when you're first booting you don't have a great set of peers and if you get eclipse then you'll start from a bad place and you can never really correct from that uh and then second is because it's just um transferring that large state over the b2b network has been it's been a bit of a challenge and we haven't really got to it we'll see quite seeing the need for it

Łukasz Rozmej: so actually never mind we have similar idea that we can optimize some things

around this in terms of uh syncing and we actually bake in that hash into our conflicts on every release and but release fairly often like every two three weeks so yeah uh that's that's how we handle similar thing we're just baking for every supported network

uh actually yeah that

Paul Hauner: that has also been discussed in consensus land but it just hasn't since the slayer but it just hasn't um happened that way might be a good kind of discord conversation

Andrew Ashikhmin: yeah because otherwise for mainnet it

will be prohibitively slow like i i strongly think that for the mainnets checkpoint sync should be the default like whatever is used in fura baked in hashes or whatnot but it should be the default

yeah if i would have it baked and then even the user wouldn't specify something new where you would have a fallback right

Péter Szilágyi: i think baking in a checkpoint thingy is perfectly valid i mean we get has been doing that since forever for the live client um since you whenever the assumption is that whenever somebody downloads a new get client i mean they are trusting us i mean we are handling their keys they everything so if they don't trust the code then why

even right so if their trust is there then the code can actually ship something uh reasonably new the only downside is that it's kind of a pain in the ass to update these checkpoints every two weeks when we do a release and and also if you have a a lot of people what they do is that they upgrade gets when there's a hard fork and then they don't touch it onto the next hard fork which means that it might maybe there's six eight 12 months passed in between so if you try to use the 12 month old client to sync with an old checkpoint then it's going to be hurtful so in order for this developer to sub checkpoints to work people kind of need to run the latest code at least when they see

and i still think pointing the beacon clients to interior is a horrible idea that's going to backfire

Łukasz Rozmej: so like i said i will use this way that i would take in some defaults at the point of release and allow people to override the course with something

new yeah that seems the same as the version okay anything more

Tim Beiko: on checkpoints okay uh but yeah obviously uh we can

continue this uh in the discord and uh yeah encourage people to check out the documents that terence linked in the chat uh where dany has uh gone through also some of the the different approaches um cool uh next up uh mikhail you wanted to go over some issues around uh latest valid hash um you wanna give a bit of context on that

Mikhail Kalinin: um yeah uh so latest well attached we have been discussed it like a while ago and uh apparently there are some difficulties in supporting it at least from what i heard at least in guess

and yeah i i just want i think it makes sense to start from like somebody from i guess explaining what's the difficulties they have to support it as it is stated in this back and then just

see this conversation

Péter Szilágyi: i think uh just a quick recap uh just to make sure that we're on the same page the idea is that when we get i don't know new payload or set for choice update the idea is i mean the stack states that we should if the beta client tries to push us onto an inbound chain for example get nodes mined a really long invalid

chain and then the nethermine client tries to feed that to the netherlight execution layer then the expectation is that the execution client starts processing this long side chain and then halfway through where it hits an invalid state transition it just stops and returns that okay block 500 out of the 10 000 is invalid this is the theory now the practice is the problem is that this new payload or fortress update we are expected to return instantaneously to it for example we return that okay we don't have this chain yet so we return syncing that's nice and all we have a 10 000

long chain which halfway through is broken so we say okay sinking but by the time we discover that the top way through broken we already replied to the beacon client so we only know that block 500 was broken and then let's say one minute later the beacon client says that hey here's a previously it gave us a head to block ten thousand and now it gives us new block times ten thousand and one but again we have absolutely no idea what this block 1001 is so we need to start a synchronization up then we need to go back all the way back and then eventually we will reach the same decision that okay block 500 was bad

but again we had to reply much earlier so we never have the response available to tell the beacon client that a block five thousand blocks ago was invalid so every time we figure out that there was a bad block five blocks ago it's already too late to to tell this the to the beacon client and the only way we could work around this would be to maintain an entire chain or set of bad blocks and for example when i the beacon client tells us hey here's 10 000 blocks and the 500 is bad then i would need to start tracking everything from 500 upwards to 10 000 as bad block

and then whenever i get a new fortress update i somehow need to link it into check if there's there was this in this set of bad blocks the parent is present there if you assign it to someone figure out which was the origin of the bad blocks and in general it just gets a bit messy to maintain a big pool of bad things because uh if the client works perfectly everything is nice if there's for example a data race or something in the client then you could end up with simply blacklisting the canonical in the good chain because and if this actually happened at one point with david three four years ago the snapshots had

the bug and very rarely a block was rejected for whatever data is but if you try to import it a second time it succeeded so nobody really noticed it i mean we got issue reports that sometimes we got a bad blocks but the network didn't care because everything just continued but where we so if we had a bad block blacklisting within the client implemented then actually every single time one of those clients had this data race issue it hit this database issue they would have just dropped off the network essentially gradually all the nodes dropping off the network because they would blacklist the main chain and this is our our concern that it's not trivial to maintain it and track it and anything

that if if something goes wrong then it can have really really bad consequences and for example this is the primary reason why get does not have any form of reputation management doesn't have any form of bad block tracking penalties nothing the likes and just another story um probably not many people here were present during ethereum's olympic test networks but at that point geth did have blacklisting every reputation management and everything on on the network and at some point the entire network just fell apart into thousands of side chains and eventually every single guest note was on its own

little side chain because there was some bug in the bad block tracking and everything every node blacklisted pretty much everybody else on the network so every miner was just mining their own little chain it was it was a horrible show and in the end the solution was actually to just remove all the blacklisting and just try to be as robust as possible to bad things and try to reject bad stuff as fast as possible but uh but not track it beyond processing the request so that's a very long background

Mikhail Kalinin: um so the risk is if we have this blacklist and

in particular same gap so the risk is to uh just mess up with it to have a bug in it and probably say that something like uh on the ballot chain is actually invalid say this to see our client it makes its own decision suppose i mean did i get it right from your description so so

Péter Szilágyi: pretty much two things one of them is tracking the the bad blocks or bad chains is very expensive we if we do it in i mean we could do it in memory i i'm not sure what how attackable that is or not if we have to touch the disc

then it becomes super messy and ugly and since tracking it is not necessarily super obvious [Music] any mess up can have

Mikhail Kalinin: really catastrophic consequences

Péter Szilágyi: and the question is what's the point why why is it so important to be able to say that when you give me a block i should be able to respond that five thousand blocks ago the chain was bad

Mikhail Kalinin: uh yeah that's uh that's like another question yeah but before answering this question um one thing i was just my naive thoughts was just uh okay so the client has uh it the client has all this like data it's been fed by ciel

and it follows eventually in value chain i mean it current there's an invalid block in the middle of the chain right but it knows what the tip of the symbology chain is and uh yeah the naive implementation that i've been thinking about is like okay so we found we encounter a bad block uh then we take this bad block ask the downloader whatever component is responsible for tracking the not yet synced but sinking in progress chain find the tip of this chain and okay store this tip and this bad block like a kind of a map and just in memory uh just in kind of a memory cache not not persistent to disk and then when

next time uh dcl uh sends you an uh sends a new payload uh check that the parent hash uh and the actually the block hash are not in this uh like invalid cache right so if it's an inva if it is in this cache of invalid tips like of tips of invalid chains then respond back with this information with the status invalid and also add this latest valid hash uh which can be like stored in this case as well so that was like naive uh approach native thinking on it

Péter Szilágyi: so one issue with that name approach is that what happens if

for example he you give me a head and for me to discover that there's a bad block it takes me 10 minutes because let's see i have to sing from genesis and then i i actually did discover that there's a bad thing i blackfist your tips so to say but there are already five sorry not 5 000 maybe 200 new blocks on top so next time you give me a fortress update it's going to be um there so essentially if there will be something in certain cases there could be gaps between the stuff that you give me so me blacklisting the tip is not enough because it won't

uh the parent might not be in this tip set so yes when i when the downloader starts processing the download the world of 7 also has to be aware of this blacklist and constantly cross-check blocks against this blacklist and that's exactly how

olympic that fell apart

Péter Szilágyi: that was the exact implementation we had there

Mikhail Kalinin: yeah yes right and yeah the assumption for this implementation work is that cl will be sequential in what it gives to el if if if there is a gap there is no way to like to link this to fill this gap right and to respond back correctly so that's what's just a kind of assumption and we assume that

basically cl uh always sequential almost always sequential of course there could be failures uh but if there is a failure so it will just start syncing uh again with this chain right yeah we'll encounter this uh bad block once again and like second time uh we assumed that cl uh if it failed previously yeah yeah so you just should converge at some point

Péter Szilágyi: so the i think the catch there is that there's a so you announce uh ahead i mean the cl announces ahead of you start syncing and then when the cl announcers and you had while the execution is sinking how do you handle that do you just move then

you have i mean do you just update sync to the new i mean that's what get does i have no idea what uh what other clients do so it's um it's not super obvious and that that's a very specific implementation detail and relying on all these little implementation details is a bit going pushing it a bit far i mean it won't be as robust as it should be in my opinion i see um okay i guess my question is so at the end of the day everything is solvable and the question is how much pain it is and how

why we need it so if there is some super strong reason why we must do it then i'm i mean we probably can't figure something out that we feel comfortable with maybe it won't be super robust but something that we kind of can't sleep with while at night with but it would be nice to know why it's so essential

Mikhail Kalinin: yeah yes and yeah that's that's the like i think at the essence of this discussion okay so why it's why it's important uh so um there is a risk uh there is an attack vector um basically uh the risk of this attack vector is actually very low it comes in

a very very particular network conditions when say uh network is not finalizing for some period of time and adversary has a portion of i don't know owns a certain portion of steak or there is a bug in the folk choice or both or whatever so there are some very very particular conditions uh in which if uh el clients do not respond do not support this latest ballot hash as per the spec currently the whole network uh like the all validating notes let's suppose that neither el is supporting this correctly

right so it's always returns thinking uh if uh cl sends uh keep keep following this invalid chain so and if all other data nodes will be forced uh into the fork where there is an invalid block somewhere in the middle we get the network uh the network of uh validators that are in never ending syncing loop and this is a kind of like wiredness failure that will require manual integration intervention to get recovered from but once again it comes with like a very very particular network conditions it's like not something that

should happen ever on the main net but still uh so my intention and why i'm like uh like my intention of having this implemented across all the clients is that we just you know remove this vector from the table and don't care about it um so that's that that's that's the reason why we should why why we have to support why we must support this like i i i will not like go in the details and the weights of this like uh attack and stereo like and the conditions but we can do this i mean i don't want to spend time here um like

discussing it we can do this offline in discord or whatever else um lucas was your sorry lucas has had his head up for like 10 minutes uh was this related to the same topic yeah okay

yeah go ahead please

Łukasz Rozmej: so oh i just want to say that we implemented this nevermind i think we still have some bug or two when when the syncs with the sync scenarios so don't don't treat it as as final or production ready or anything but we definitely do something like mikhail explained that we have a very simple um

cache in memory i think currently it's 512. when we just keep a hash of the block and last valid hash for that block if there is a detected that is wrong so when we think when we are thinking uh as when we detect invalid block we go through all the uh [Music] descendants of the block and mark them with that last valid hash so we are keeping the tips of the chain marked if it's longer than our cash we're keeping the tips of the chain mark not just one tip but like potentially up to 512

blocks in that tip which we considered good enough so when the new one comes build up on this on this one of those blocks we will be able to respond with less valid hash so uh if the consensus layer is let's say sequential enough uh we will have eventual consistency here that we will be able to respond with as valid hash but um yeah that's that's how it's being been built the one big potential issue is with state sync if we stay synced and

the problem is with the state in terms of why the block is bad like the state routes doesn't matter the state we want in some of previous blocks we we won't catch that that's i think beyond the uh scenario in mind that we cannot really detect involved hash with state sync with snap single fast yeah but i mean that's that's kind of the low limitation of sync work yes yes you start so every the assumption is that when you finish the first initial sink then the head block was fed to you was valid yes if that doesn't hold then i mean all bets are off and you're on the wrong

chain and there's no coming back but yes with these initial things is that you it's uh you do it only once when you start up your note and never again so it's extremely hard to attack it because somebody needs to be listening exactly at the correct time and on the correct network prepared with the exact attack vector just tailor-made for that specific chain condition so it's an extremely unlikely

yes yeah

Mikhail Kalinin: and yeah this this like risk of this attack is only uh this kind of attack makes sense only on uh fully synthetic online notes so it's

definitely what we would like to prevent is only related to the block sync when we when the client is executing blocks not the state

sync another question

Péter Szilágyi: what happens so um let's say uh you feed me an invalid chain i remember that this chain was invalid and the last valid block was 5000 blocks ago the next time when i when you tell me to import its child i will tell you that okay the last um i mean 5000 blocks ago we got a bad block what would what would be the reaction of

the consensus client and specifically more interestingly what would happen if i would add a heuristic that once i returned that this chain is invalid i will actually forcefully forget that it's invalid so if you try to feed it to me the third time i will again do the processing for the fourth time i will report it simply for the fifth time i will again do the processing for the system i will report it's invalid the idea would be i would tell the execution client sorry the consensus client if i if i consecutively twice hit the same issue i can actually tell it that okay it's i'm getting the same issue but i would actually give myself the chance

of reprocessing it and maybe getting out of some weird data race or some weird scenario so

would the sales client choke on such a behavior or not uh so uh the consent slide

Mikhail Kalinin: will uh remove this chain from starting from this block from from the fork choice i guess and from from the disc if it was persisted and will rework to the other chain which is valid um that's the expected behavior i don't know if like uh in the next time when i think it's implementation dependent that asked to

consist the client developers uh if if this same chain will be uh given uh like once again to on the consensus layer will will it follow will be consistently a client start following this chain and do the same actually um i mean like encounter the same um yeah but i guess what what happens in this case is that execution layer client will already have a state right and yeah if if the same block is given once again then it will just be invalidated instantly

full

Paul Hauner: lighthouse we would um

if if the el told us that it was invalid we'd remember that it's invalid we'd we'd um mark that entire chain is invalid and we wouldn't try to apply anything to it again um and any time that we were told that it if we were told then again that it was actually valid um we'd log an error saying that that you know you told us that it was invalid but it's actually valid now so there's a consensus fault

yeah so my expectation wouldn't be that i will tell you that it's valid rather um i i have to decide i'm really afraid of tracking blacklists because my experience is that they can sometimes they go horribly wrong so my my idea

here now was that if we were to add some form of blacklist then ideally i would add a super aggressive exploration or addiction to that blacklist so that it would get thrown out very very fast so if you so if you keep trying to feed me the same bad stuff over and over and over and over again then occasionally my i would like to be able to just say okay since you're pushing it so hard let me just redo it that's nice to read recheck maybe something went wrong and with such a behavior i could accept such a behavior because then if there's some data race or something

like that there is a chance to to somehow escape from that stop state so so

Łukasz Rozmej: peter and catch what you're saying but uh we are not for example blacklisting nodes but we are not listing blocks that we kind of failed on validation or processing of validating the state route do you think this is recoverable in any way that's i don't see it as recoverable

Péter Szilágyi: no that's actually happening again so that as i said about four years ago there was a bug in gas where there was some uh i think either when i implemented the try the pruning in

memory pruning or only implanted snapshots there was a point where essentially it's very very rarely blood to get rejected because something else was also mutating the try or touching the try and if you try to reimport the same block then you succeeded i think it was the blood processing concurrently with pending block execution so there was this weird scenario where two executions were running at the same time and corrupted the

tribe

Tim Beiko: but if you actually

Péter Szilágyi: tried to reimport the block it succeeded because it was super rare for that to happen

okay okay

Łukasz Rozmej: that's a valid scenario for a bug i fixed you a few other things like that

and never mind already so i see what you're saying i'm not sure if we could recover as easy but maybe we could

Mikhail Kalinin: um yeah we should probably uh discuss continue this discussion uh some other place but uh before uh like we uh we'll finish with the latest value here we have a question to berser and aragon do you have any difficulties in implementing this as well

Andrew Ashikhmin: we have implemented some basic caching so yeah you have a cache of invalid blocks and for each embedded block you have the the last letter hash but

it's kind of it's not very elaborate so maybe it makes sense uh we have a testing call scheduled monday uh to use that to like focus on this some more and obviously we can we can use the discord like in the in the meantime but um yeah it seems like clearly uh spending more time discussing this on the call is is probably valuable

if

Łukasz Rozmej: if i can have one last comment here um of course yeah yeah because it's in memory never mind so restart if we like blacklist something incorrectly the restarting would uh erase the blacklist starting tonight so that would be a recovery scenario

all right

Mikhail Kalinin: yeah and then also uh elsewhere and uh yeah one of the options is [Music] somewhere soon after the merge because as i've said that the um like risk of this attack is really low and it comes with like very particular networking condition and so forth so we can probably remove it like from the

critical path to the merge uh so uh but i would strongly uh be in favor of leaving this in back as these and also yeah the hive test that we have if we if we collect keeping this requirement in this pack so uh we have the hive tests uh in place and just accepting that some hive tests that are related to this particular issue will just fail for the clients this is the one one of the options that we can take okay yeah i think that makes sense yeah so let's yeah let's continue this uh conversation on discord and then uh on on the testing call monday and um and yeah we'll follow up from there but

uh yeah i don't think uh this is something we're going to resolve in the next like five minutes um yeah thanks thanks mikael for for bringing this up and and the client teams for for sharing all the details uh related to the implementations um any final thoughts on this before we move on okay um so i guess yeah the the next thing we had was basically uh sepolia and how we feel about uh about moving forward uh there um i guess at a high level uh the sepolia beacon chain is live it's run

through bellatrix so it's basically you know just waiting for a ttd to be to be set um septolia is also a bit different from roxton and gordy in that the validator set is is permissioned uh kind of like gordy was uh pre-merged or sorry gordias today on on the execution layer um so it's much easier to coordinate like a network upgrade there um because yeah the the node operators are uh are kind of large entities that uh that that we can reach out to um yeah so i guess i'm curious from client teams like um i know on the cl call there were talks about potentially uh merging sepolia

uh sometime in the next week um yeah i'm curious to hear from like el client teams what uh what do you all feel is like a good timeline to to to merge the polio yeah uh lucas

so

Łukasz Rozmej: we are planning a release with some fixes next week um it would be great if we if it was after that release so we can enter that but so either late next week or or week after that is fine for you

early week after that is fine for us but yeah that would be great if we could have it yeah we can curve from uh from bay

suicide as well i think our consideration is that we we don't want to overload uh the end of next week uh seeing how great glaciers gonna be going out but yeah either before or after with a preference slight preference or after okay uh guess aragon yeah forget

Péter Szilágyi: it's just about setting the ttb and doing the release we can do that whenever it's just our request to be deaf it would be nice to give people a heads up of let's say maybe a week that okay here's the new release run this for sephora okay so if we have a ttv then from my

point on we can just release the same day or the next day and okay

uh ergon um yeah

Andrew Ashikhmin: so the same in aragon we can make a release relatively quickly so no no hard preference

Tim Beiko: okay in that case um would it make sense i i don't have a ttb value now because i wasn't too sure of the date but would it make sense if uh monday uh i can i can come up with a ttd value um and then we have client teams put out releases like ideally around midweek so if it was like

wednesday so that like thursday we can announce it um and then we we can we can fork uh sometime like mid the week after that so there's like roughly uh a week um i went for by the way the the next week sorry two weeks from now on the fourth of july is a big holiday in the u.s so we probably don't want to forge on the fourth of july um or like yeah so maybe like you know in the couple days that just after that yeah

uh clear

Péter Szilágyi: yeah because only just a quick note uh last week we also configured the dns discovery for sepolia so

i don't know which clients support it i think many clients many insufficient clients support it but the tldr is that you can also point it disappointing to run discovery and it should work and it should actually help a lot with fighting peers because previously it was kind of annoying to find support yes i i complained about this on twitter and everyone was uh sending me their e-notes my dm so glad to see that this is uh going live cool yeah okay so let's do that then so um by monday i'll have a suggested ttv uh i'll share that in the discord get like

a plus one from all the client teams um and we'll aim we'll do like we did on on robson basically the ttd will be like very high such that like it's unlikely we would hit it um ourselves uh or sorry exactly we would hit it with this rate but then we'll rent uh some hash rate to accelerate when it actually gets hit um and so yeah we can have this value shared uh on on monday to teams and um teams have a release out by like next week late next suite we announced this and then uh mid week after that like around the july five or six uh we

we can have the the fourth happen on on sepolia

um anyone have it yeah i have a request

Péter Szilágyi: if we're going to do the same thing that we did for upstand uh please uh so with robson the whole mining was a bit of a pain because essentially we the hash power pointed to the mining machine just overloaded the mining machine i think it overloaded the mining pool and yeah i think mario was saying that essentially there's not a really no really good open source mining pool so it's kind of the expectation is that you won't be able to throw too much hash power at it without it choking so um

trdr let's not rent out 200 gpus and throw all of it and then wait three days because whole hell breaks loose so yeah we can maybe yeah we could maybe also like start because we haven't been a heads up too we can start renting some gradually as well so yeah it doesn't have to be all as close and it's like if we rent some and then we see other people are mining we can obviously turn off our rigs so yeah we we can definitely uh try to yeah make it a bit less chaotic

this time around

Péter Szilágyi: or alternatively what you we could do is to i mean running a sephora node and probably mining pool is fairly light so maybe you could uh if we want to rent out

a significant number of um gpus and maybe you could spin up either two three four mining vms and point the rings to different vms so that they lighter than super overloaded

Tim Beiko: right yeah yeah that seems like something that's

probably only renting one of these vms costs i don't know half a dollar an hour or one dollar so it's not really relevant for yeah that's not the cost

Tim Beiko: yeah okay yeah that makes a ton of sense um yeah and uh i might ask mario to ping you uh peter to chat about that some more um but yeah definitely something we we can do a bit smoother than the last time um

okay so yeah that makes sense so let's aim for like a merge date for separate of july 6th more or less uh plus a minus a day and then um having client releases out uh by this wednesday ideally so we can announce things on thursday um yeah um cool anything else on sepolia okay and then one thing i just i did want to get back on back to from earlier is shadow forks so like um we have obviously the mainnet grey glacier fork oh sorry peter

is this about

sepolia

Péter Szilágyi: yes just a quick question

yeah

Tim Beiko: go for it

so with sephora

Péter Szilágyi: there was a discussion on how to avoid the not enough ether issue that appeared on gurley and just the question was that solved on the beacon client side all right like yeah yeah so that the i think i think what pair you got you want to give that up right there yeah i can uh yeah it is solved on the beacon client side essentially what we did was um set the balance value of each and every validator to be a million at the genesis state and it looks like that inflates the supply by about 1.5 billion so eight

times what it is and there are a couple of other nozzles that we can turn to ease the pressure over the cup over the coming years uh i made a comment on that on github i'll try and find that view because that has a more concise explanation

cool

Péter Szilágyi: okay then uh i guess the only catch is that we need to um to survive with the pre-allocated the genesis allocation to the teams until then so yeah for public faucets uh essentially i think every team was allocated kind of 100 mil or something like that on during the greece meetup and maybe everything could uh contribute a bit to

the faucets yeah i've been i've been doing that by the way so i've been sending a couple millions of polio ether here and there to various faucets and people um so there will be more and more faucets like spun up in the meantime uh but i think i think we'll survive with the genesis allocations uh until we basically have withdrawals shipped on sepolia and then from that point we'll get basically infinite inflation from from validators or very high inflation from validators cool and yeah perry share shared a link in the chat about this um

okay uh yeah then yeah the next thing i wanted to chat about was uh shadow forks so if we have greg glacier uh next week and we have uh sepolia the the week after do we also want to schedule this shadow fork in the next couple weeks um i don't know perry do you have thoughts on that i haven't thought of that my thought on that is that yes so since uh the shadow frog from yesterday was an epic fail uh i think the fact that we are pushing them the test nets shouldn't mean that we should skip on on pushing the maintenance since i mean um pushing the desktop isn't too much of an effort at least from a client

developer perspective probably from paris perspective it is but still i think as long as we have there are no issues with the main shadow force we should hammer it and not wait okay and i guess then the two things that would be neat to do on the next shadow fork is one run basu uh not in bonsai mode and two like manually set piers uh or statically set piers on aragon to make sure that they have some uh some shadow fork pier um i guess yeah that's a question to you perry mostly like what's a realistic like time for us to to get that

set up i

Justin Florentine: i think basically here would prefer

maybe a mix of forks and yeah we plan on having fixes in in uh in that next build for whenever perry's ready okay yeah i can work on conflicts on monday or tuesday usually the notes take a few days to sing because i mean that and we can have the shadow for probably the monday or tuesday after so yeah so we'd have the shadow fork basically at around the same time as the the the septolia fort like maybe a day before okay the same week instead yeah cool yeah that makes sense um peter is your hand still up

for something else are you oh okay perfect okay so yeah so expect a shadow fork uh not next week but then the week after um okay we only have 15 minutes left uh and there's like two things left on the agenda so there's the debug one for my client uh but i know so paul had one about the awkwardness time and i want to make sure we get to that because it's basically middle of the night friday this saturday uh for paul um so yeah um anything else i guess before we do that anything else just like on the merger shadow forks otherwise i think we should we should go over

paul's point and then wrap up andrew

Andrew Ashikhmin: uh yeah i have one question on the merge i think um in in guess like like i've raised this issue that some ico clients use net network id instead of heath chain id but by the engine api spec we should only enable [Music] expose if methods so but but the problem i i i saw on on this chord is that in in geth for some reason is chain id doesn't work before if one five five transition but can we somehow solve this issue because i mean

we could explore either make youth chain id available from the start or decide that okay just let's expose net network id

from lighthouse

Paul Hauner: perspective we're using um net id at the moment and it's just kind of like a mistake we're getting rid of it um we also come across gef not supplying the chain id for a while but we've just kind of worked our way around that i'm not sure if there's any other cl clients that are having different troubles um but i guess for lighthouse things are kind of fine as they are

Andrew Ashikhmin: okay

Péter Szilágyi: yeah i mean um i think the idea was that

um before you have a chain id so i don't know i i this is probably more of a legacy thing that when when the whole 390 was introduced and uh before the fork it just returned nil because there was no chain idea that after the fork it started returning the chain id and probably we just went with it

yeah

Péter Szilágyi: i guess the question the interesting question is what happens maybe it was also a bit of um trying to guess the future will we ever change the change change the chain i leave probably not yeah i think it's kind of a semantic

philosophical question whether we should we return the chain id before it or not we can probably return the chain id from from the

get-go it's probably fine uh mikael

Mikhail Kalinin: yeah just quick thing uh there is a pr into engine api uh el clients under certain conditions may skip the fork choice updated uh processing and this pr makes a little bit of change and uh in terms of like making it more precise these conditions please uh you know my devs go and check this pr and if your el find how it's currently implemented it's like broken by this change just

leave a comment that's all yeah otherwise it will be emerged soon into this pack okay um anything else on the merge okay paul the floor is yours

Paul Hauner: oh yeah sure thanks um so i was just pointing out that um for a lot of asia pacific this call is kind of at times between say 10 p.m and 2 a.m on friday night so that's places like china japan singapore taiwan australia new zealand

i think that it's not a great time for in terms of work-life balance for people it's just kind of you know friday nights for socializing saturday mornings of family um i think also if there are any times for bad decisions to be made probably between 10 pm and 2 a.m on a friday night um so not great for decision making yeah so i thought i'm just bumping it back um say like 24 or 48 hours or something like that would be a huge help another point as well is that um for places in asia pacific um so this is friday night so the next working day for us is in two days so actioning things the next day is quite hard um it's also quite difficult um to like if you want to listen to the recording the next day you need to either do it on the weekend or wait

until the next day um so yeah i'm just proposing the weekend i mean that's i suggested pushing it back 24 hours someone else said do it 48 because people like to drink on thursday nights um i don't know i'm not really sure people like to drink all the time but i think at least um friday nights are clearly you know should be kind of sacred times in the five-day working week so yeah that's my proposal same time but just 24 48 72 hours earlier please it's directly yeah thank you um and yeah and worth noting obviously the cl calls happen on thursday basically um so that's a natural kind if it would be nicely aligned if like this was also on

thursday because it's like people know every week on thursday there's one or the others um yeah i guess you know the main stakeholders here are client teams like i'm curious to hear from like the different kind teams could say thursday 14 utc work for you okay so marius this is fine and i guess does anyone on like a client team feel strongly that like this is not a good idea

the

Marius van der Wijden: only thing i really like like like about this timing is that afterwards i

i can go to the weekend and it's like a good end of the week oh yeah i could just end my week on first you can still go to the weekend after 1400 utc uh on friday but yeah i i agree it's a nice way to wrap up okay it doesn't seem like any objections um i i personally prefer like thursday is my favorite proposal like one because it aligns um one because it it aligns with the cl calls and two because it does give us like one next one uh day after to follow up on stuff like i also tend to find

that's like an issue because i pay everyone after this call about different things but it's basically already friday night in europe and so uh people will respond my messages um so yeah i'm i'd like to do that um i think what my my like preferred probably way to do this is to just like have the next awkwardness be the last one ever on a friday and so people kind of get the uh a heads up like we typically try and not like change things like if i don't know if somebody in the next two weeks comes up with like a really good objection for this um i think it's hard to leave them some time but um yeah i would basically make the next

awkwardness the last one on friday give a heads up that like after that it'll be um it'll be on thursdays and uh and then yeah have it at thursday 14 utc last call if someone has an objection [Music] okay well yeah thanks a lot paul for coming on uh yeah um yeah let's do this

Paul Hauner: yeah thank you very much much appreciated really do appreciate it cool

um and uh last thing on the agenda like client you had a proposal to standardize debug name spacing across uh clients

yeah just a quick thing um does this sort of come up in a couple of these like in person um interrupt situations where we're trying to test why different clients are having different issues and we've talked a bit about having like certain rpc endpoints to retrieve information i think in amsterdam at devconnect we talked about maybe having one to get the raw receipts from a block and so just like over time there's been a handful of these different endpoints that it would be nice just to have as a like standard across clients and so i have a proposal to make four of those standard under the debug name space and the idea would be that we you know

each client just implements this um these rpc endpoints and they're all conforming to a spec we have this open rpc spec if you haven't seen i'll post a link to to the pr and that way we'll actually be able to run tests against these against these endpoints in hive to make sure that everybody conforms to it and i think this is just you know not only is it nice having standard interface across clients but then we can build some more tools on top of those interfaces to help debug clients without having you know individual behavior for each endpoint so if some el teams could take a look at that proposal and just like let me know what things you've already implemented and

you know what things you would like to see implemented or maybe you don't think makes sense in this like super easy debug name space that would be really helpful are you looking

Micah Zoltu: for feedback on like things that would be useful or should it be constrained by things that would be useful and are relatively easy right now i'm i feel like it was better to go for things that are useful and relatively easy just you know i i i don't want to put more work on people but it would be nice to have this as soon as possible just so that there's you know less interface people to worry about if we're trying to debug like more of these issues in test nets um and so that's kind of why these ones that we

have are just basic getters stuff that like a lot of clients already have but useful ones would be helpful to know as well i would open an issue on the execution apis and we can talk about it there

andrew

Andrew Ashikhmin: uh so it says that uh geth uh supports all of these uh so micro like why can we simply use the intel the get current guest interfaces as the basis what's is there a reason to deviate from gets interfaces um i mean we can but there's the uh two the the get header rp and get block rlp only take in like a json integer which is just different than all

the other rpc endpoints they take hex strings so i went ahead and took the liberty of just making that conform with the other types of numbers that we take in

yeah

Péter Szilágyi: so i guess with the with the specification the idea is that all the clients can just tell what they would like to see and how it would they would like it to work so it's not just get enforcing it's so these methods that are in gas we just implemented the way we did because we whenever we developed something we did it and we just hacked it in they weren't

particularly well thought out and now if you want to standardize it then it would be nice to actually put the ten minutes of thinking behind them before the blind implementing whatever

guest has

Łukasz Rozmej: uh i have a comment about debug get bad blocks because it's not trivially to implement correctly so one implementation that i think we have in least pr in never mind but we haven't measured yet is um just to keep it in uh memory but then if you don't have the bug enabled module you have to restart it's the node to the enable it potentially we can add json rpc on admin module to enable debug module but again

admin module is also disabled by default so when you restart you lose this cache and you lose these bad blocks potentially you could store them in some database persistently i think kev does that but then again you need to prune them potentially which is like creates a lot of complexity i'm not entirely sure if you want to have it in

a standard

Péter Szilágyi: so we do not store it in a database as far as i know we just uh store it in memory uh the only thing is that so if you attach to get on the ipc that all the apis are available always so if it's not exposed on the rpc worst case scenario i can just attach the

docker container directly i'll pull it out from there

okay and the ipc is always exposed rpc is always exposed unless you specifically close it down is there a reason nethermine doesn't expose debug by default

Łukasz Rozmej: yes if your endpoint is public then you know it can cause a lot of load on the node

wait but i guess

Péter Szilágyi: the question was uh why not expose it on ipc so http and websocket gap is also very restrictive but on this unix socket we just i mean if you're already on the machine

within the docker container

then it's already game over yeah true

Łukasz Rozmej: be told we added ipc only like a year ago and didn't really think about configuring it this way it's disabled by default so you might want to enable it by default and expose all

apis there

Micah Zoltu: my uh we can talk about this somewhere else since we're closed on time but my initial reaction is i'm unclear what the attack vector is where having debug exposed publicly is more problematic than having the um exposed publicly like if someone wants

to take down your node that's explosively just hammer pretty much any endpoint and you're gonna go eventually if you have set head exposed publicly then you could just

arbitrage

Micah Zoltu: sure sure but said headed is crossed out i'm screaming for this exactly oh okay you're just saying yeah the ones that we have provided the getters i think it's i don't see a problem with it but i see like with get bad blocks that's a little bit more dependent on what the client sees on the network and so it's possible that if you're trying to run a like a proxy in front of several clients you might have different responses based on which one you route the command to

you would have to go ahead with it sorry you would have the same issue if

you have two clients with uh mismatching chains that every time you request a block a header or you could get different answers so i think once you start hitting debug you don't want balancing in there

Łukasz Rozmej: yeah like it's like mentioned there is a set head generally available so we might want to like either move that or like admin module or something and if we want to make debug public we need to review what's there and potentially remove some things from debug module s yeah the the list of methods is just

four methods right now it's in the link that i sent

yeah and i would definitely only advise making debug public if it is getters only um if there's any writing then i agree it should not be public yeah

Péter Szilágyi: i think we kind of have we all agreed and discussed that the debugging space should be just a read-only thing so that you don't have to care and marius had a really nice idea to what to call the chain modifier namespace you can call it a node and then you just a node.seth okay

um okay i guess we're basically a time um map i guess yeah people can can go to the the pr there is there yeah that's probably the best place to keep discussing yeah that or the jason rbc api channel discord i'll be watching both okay cool um any final thing anyone wanted to bring out bring up pretty quickly um okay if not the last thing i'll just share is like the two eips that uh people wanted to discuss uh that were

moving to the next call uh there was an update on four four four four and um five zero twenty seven so those are both linked in the agenda if people wanna have a look uh and we'll try to uh to get them uh on the next awkward apps um but yeah thanks everyone for coming

thanks thanks thank you bye bye thank you thank you bye thanks bye

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.