All Core Devs - Consensus #79

Transcript

[Music] [Music] [Music]

[Music] [Music] [Applause]

[Music] okay the stream should be transferred over here is the agenda if you are in the youtube chat and you can hear me let me know people might still be trickling in um all right cool so first half we will discuss anything related to kintsugi and the merge then go into any other client updates if we have them then some general discussion technical discussion about research spec and other things beyond the merge and open

discussion closing remarks at the end um thank you alex and thank you tim for handling the couple of calls in my absence i was off a lot of the last month and in fact i am still waiting through everything and catching up uh but i will get there let us begin we can start with um general consequence updates is perry here or raphael raphael and perry um hey barry people are trickling in um the

first thing on the agenda is just general kinsuki testnet update i see that we're finalizing do you want to give us a quick on that sure um yeah after the issues caused by marisa's father we were able to find quite a few bugs over the week and the chain wasn't finalizing from last friday morning till about yesterday evening um so essentially we just pruned off a couple of the forks make sure there's one um major fork and we have i think almost every client combination working um i think there was nethermine nimbus and nethermine prism or something like that that marek said he'd look into

and put reach up to the teams but otherwise we seem to be good to go and as a side note i have a bunch of databases from the non-finalizing period um i will upload them and just share a drive link there's also a dump of the proto array um i think a couple of log files and yeah there's quite a lot of debugging information in case people want to still dig into stuff great and i know that i'm always beating this drum i know we've captured a couple of uh test cases from this

i think some more should be captured and maybe make their way into hive or some sort of like environment that actually utilizes the engine api are we cataloging these anywhere um not yet but i will start a document to do that i wanted to work on a proper incidents report so that we have like a timeline for everything that went wrong um i guess that this is a nice candidate to tag on to that yeah so so mario is already adding them to his pr for um for hive okay so the ones that we that we found

excellent um i i'd like to add here that this block hash issue um partially appeared because it's not that clear from the spec that the block hash must be verified and against the block constructed from received payload uh there is a requirement for execution layer clients to verify the block structure but in this particular setup it's not that clear i mean in this setup with the separated cl and your clients it's not that clear how this struct how this verification should be done and this is what's what's bending

uh let's bend and change this back also um what which is even less clear is that this check must be performed even if the client is thinking which is important right so the way that's going to manifest in a spec clarification or i guess minor spec changes like anytime there's an inserted payload there's pretty much like a well-formedness check you know is does the blockass map and maybe some other like minor things before even deciding to do syncing or whatever and um if it's not well-formed it's invalidated immediately

right and i think that any um walk header verification including this block hash verification should be performed if it's possible there is enough data for it um should we already have to check whether this reverts finality there or should we have this somewhere else um i suppose if so the consensus layer client shouldn't be reverting finality but if it were on a branch that you decided you should never execute because it was signaled as finalized and you did deep pruning um it does seem prudent to just say i'm not

going to do that in some way so um that might surface is invalid um but we should i feel like that's a bit of an abuse of the semantics so i'd like to think about it a bit more i suspect you'd get that every time you wipe the database of the cl and keep the el running it start again right but i guess if so you can get a bunch of like weird inserts even though cl presumably is probably going to get to the same result eventually well we check if we have this this block in the database and if we have it in the database then we're maybe not going to perform this check

right but if you had previously heard finality and pruned then uh we still we still know that we had this block at this point okay then the answer might just to be utilize that cache to say it's valid i don't know if you would want to tell it it's invalid if it's no block then we can just say it's valuable so that would be a no-brainer answer if the question is what happens if we get a actually a block that's older than our finality point in that case we've been we've been discussing various

that we should just reject it with the assumption that it's some weird error in some kind and if uh if the execution no if the consensus client tells us that okay here's a new finality which actually entails rolling back the previous finality then we could just say okay i'm going to sync if you're really sure that's basically where you want to get that we could go back right i but again so if you still have the block from the block tree and knew that the actual execution was valid and cache that you could still say the execution is valid because you can get to this position like if the cl was resyncing it might

be giving you some stuff from before a finality point um so anything that is on the canonical chain so if we get the replay of old events those should be kind of accepted without modifications to our current chain that would be my guess yeah that's what we do we just ignore them well we say inserted already non-block and and don't actually insert it if it matches that the the the hash and the block number matches you would also get a stream of choice

updated from potentially genesis on from the beacon from the consensus layer um you probably don't want to roll back to genesis and then like ignoring that is probably reasonable but it means your head isn't where the beacon thinks it is how how can we how can we ignore that like how how can we know that this these fork choice updates are just like replay well you can open we can always check our current header chain and [Music]

sorry can move backwards a small amount though we might revert back to the justified checkpoint in some odd corner cases i think i if you would but you would no you would never revert so you say you have a single chain you would never revert to earlier in that chain it would have to be the only reversion would be to a different chain because you always you always end up on a leaf yeah that's true fair enough so yes if you you could potentially and we'd probably need to think it through a bit more but potentially ignore fork choice updated

that is going backwards on your existing chain like to an ancestor of your current head right if you said that the head of the chain is optimistic you might and it turns out to be invalid it might go backwards what it's worth right but it might be easier to look at the finalized but that would any anything in valve would never make it into the el canonical chain because it was invalid with respect to that layer not if we snap sync past it sure i mean more in kind of the head range

but the real giveaway would be that if the finalized hash is gone backwards then it's definitely a replayed update like the head can be more complex but finalized definitely should never go backwards yes and i think it's like if if we ignore everything where the where the finalized hash is is uh like going backwards then we already have like i don't know 99 of all of all fortress updates we can ignore them and so it's uh only like the period where we're in in like this weird period between the last finalized one and the the new

um then that's that's the only like the only issue and that's not a big issue so we can replay 64 blocks or something yeah most of the time that should be a small amount like that and it's pretty common to kind of roll back that far i mean not it's pretty common one on it yeah not in terms of normal use but in terms of syncing i mean i know clients earlier were routinely doing that on every restart and you didn't notice until it went wrong when madasha went wrong but um yeah it's not it's not an unreasonable number of blocks

and once again if we go backwards in terms of finality it's going to be ignored which is reasonable but if it's like the other fork that is about to be finalized or attempted to be finalized what's the behavior should be like the fork uh like i know the uh the new finality checkpoint is on the is like uh on the other fork and the current one well if if the finality just goes sideways whether it's below up or wherever my assumption would be that okay

the beacon chain reworks to some different chain and we just follow it blindly i mean we just assume that that is a correct chain uh cn and then if you don't have like uh the state to execute those blocks because i will trigger a thing in that case if you if there become change client announces a new finality which i don't know anything about then i should just assume that okay i'm missing something here i just need to sync it right so this is i mean you'd only expect such in you know an exceptional recovery scenario in which maybe you had finalized something that was manually reorged and so

hitting an expensive sink is probably expected in such a scenario and fine yeah so it's kind of like a fallback scenario so that if let's say there was a consensus error and some clients need to be swapped out i mean the chain from underneath sometimes if you swapped out then it's healthier if there's already some functionality implemented then you don't need to i mean if the beacon saying says okay switch to this chain it's nice if the execution client can just switch to it instead of you having to manually download integration client let's uh let's piggyback on this discussion we'll do the testing

discussion after and talk about what we've been discussing over the past 24 hours in the merge channel so i think i think the tl dr here is really that the this is not the case for all execution clients i think aragon handles things a bit differently but the default is really that there's um the execution client really only executes a branch one in proof of work once the fork choice has signaled this is canonical and so only when the difficulty of a branch overtakes the canonical branch would it actually execute uh such a branch and

whereas the semantics of execute payload and fork choice updated were implying a bit differently to the execution layer they're pretty much saying i'm giving you a payload it might be from any branch it might not even end up being the fork choice or canonical but you should execute it and this mismatch in reality [Music] led to when there's high forking on the beacon chain very high load on geth and probably other clients so the semantics of these calls should probably be may

execute on inserting a new payload um to handle the case where uh such branches aren't to be executed until the fork choice comes or clients where they can execute multiple forks at the same time um and then a must when the fourth choice comes in that's i think the initial discussions that have happened but there's certainly something here to iron out because of the mismatch and expectations in reality peter yes so i just wanted to say that currently get in proof of workload what we do if a block is broadcast obviously the block just builds up on

our current chain we executed and problem solved now if the block does not build upon our current chain which would be the case here with the execute payload we have two options one of them is if we have the state for the parent then we just execute as previously right if we don't have the state for the parent then we just stash away the block in the database and wait for a total difficulty criteria between the cv right so i something like that could be done here too whereas with execute payload if i have the state then i can just execute on top and if i do not have the state but i still have

the parent then i could just stash the block away and maybe execute it only when i get a uh subject right right right so i think that i think that we do have to modify the semantics of these calls to allow for that behavior in some way and so execute like i said becomes a may uh you know it's a must if it's the head it's a must if i guess it's a must if it's building on the head really um and it's a may if it's on some side branch and then it becomes a must when you get a fortress updated which is the equivalent of the difficulty of such a branch overtaking it um and there's certainly things to think through here

uh because i have a question the semantics we have today do not really match the reality of that go on mikael yeah yeah in general i agree with changing the semantics that would allow um optionally to do the execution either on the execute payload where choice updated whichever suits better for the particular client i have a question to peter uh you've said that if you have this uh parent state um and this is the side chain you will execute a block right i'm just wondering what's uh what are the cases when you don't have the state to the parent is it like when you just like snap synced

and there is a side are it deeper than one just deeper than one block uh no uh so get maintains the state for the last 128 blocks so if your new block is rooted within 128 most recent blocks then we will have the state available to execute if it's rooted deeper then we will probably already have pruned the state and we would need to roll back to some fairly deep checkpoint from which we could execute everything forward okay that's much better than i thought yeah that's that's yeah much better

that's a lot better than i expected than i thought based on our conversations the past 24 hours okay that's that's good i just um i was assuming you know if you were at a depth of more than one from the head and you had these branches that you you're really only maintaining the head state and that was counter to what we had discussed months ago but i was just a bit i became confused in our conversations um okay that that ends up good and that ends up the change in the semantics ends up being a much more um edge case than a normal case andrew's got his hand up and i suspect he has bad news let's hear it yes so i'd like to

say that for aragon it's different for aragon we don't have multiple we don't support multiple states we only have one state and for us any kind of reorg is relatively heavy uh so we cannot simply like almost simultaneously switch between states we have to do a quite expensive reorg right okay so even if um if i have just a a two block depth at which the branch occurs it's it's you know a well it shouldn't be problematic like because the the reorg will be faster

so we have state divs so we have reverse stations we apply reverse state divs to get to the common point and re-execute the missing blocks okay so still so in that in that paradigm still changing this um these semantics to be may and less on the head and a must when you do a fork choice updated tell me this um in proof-of-work you would only do such a reorg when you get a signal that the difficulty is worthwhile on on the the side branch right okay yeah correct so that again that still even though that's different behavior or different

costs than geth that's still in the direction of the semantics we've been discussing yeah just wanted to mention that it uh get is actually going towards that direction so uh although we will still maintain 128 states always so that's something we are going to keep but we will be also implementing the reversitive approach so if for example you want to do a 10 000 block pre-org it could mean that we would need to apply 1000 universities but it's still fully compatible with all the discussions here so good to know

um can any other execution layer clients weigh in on the previous engine api semantics and the the ones we're discussing and how that might affect them or if this sounds kind of synonymous with how their client works if no comment i'm going to assume that that's generally kind of in line with how things work um i have like the last question with respect to this topic uh to gas team um once you have these reverse

diffs will you still main going to maintain the um several states yes yeah okay yes i've just missed this okay so um essentially you could think about that we would have this uh persistent state on disk that's 128 blocks old any newer change changes would be stored in ram in memory so essentially the the latest 128 disks would be environment you could have an arbitrary branching and the older this then if you want to go back more than this

first 128 block gold persistent state then you would need to start applying these reverse tips to mutate it back to where it was but we would still make i mean snap sync also relies on on the state being available for the last 138 blocks got it so you'd you'd have relatively cheap inserts in the 128 range but you'd be able to do these deeper uh reorgs without switching to some more insane sync mode yeah so essentially anything that's shallower than 128 should be instantly executable

got it so i do believe based on the conversations that are 20 hours for 24 hours we need to write down a modification proposal to the engine api to account for this um and um you know model out the interactions on a normal and and some of these edge cases are we heading towards having fork choice updated return valid in and invalid rather than just success since it's now the must execute i i i personally need to think about it

a little bit more um i that does not seem like an insane design choice but i'm i need to think about the implications more what do you think i'm leaning more and more towards doing it we we discussed it a bit um and kind of settled on well let's not change things but i think if we're going to make a decent amount of changes it feels like it it makes a lot of sense to to get the value there right if one corner case is let's say um in theory it would be possible for me to feed if i give you let's say ten blocks five of them valid five of them afterwards invalid and then i do a four choice

update to the head of these invalid blocks then what would be the meaningful return i mean the chain setting the head is not possible because some blocks and there are right and so the meaningful return ends up being invalid and that's my maybe why the fortress updated should be able to handle that because on such a branch i don't actually know if it's valid until i signal for you to execute it with the fork choice updated i'm saying that there might be some weird corner cases yeah i i agree

i agree there there might be some uh some nastiness in there and we need to be careful it's definitely worth like more thought thinking and analysis yeah but i'm also leaning towards having this execution semantics for both methods with respective response statuses and the respective behavior on cl side so it will need to understand that the choice updated also may return with the execution um statuses and so finished sinking if it's appropriate in in the current state and yeah react

accordingly to the new statuses and new semantics okay so from this uh mikael myself and some others will try to iron out a modification proposal and seek a lot of input and feedback on it um to make sure that this week hit the edge cases and um there it's works with el clients peter something uh just touching on this uh

discussion do we have some form of infrastructure or capability in place to test these kinds of scenarios and these kind of failures that i don't in essence can we spin up a non-finalizing bad state i mean a network in a bad state just to see what happens what science right what do providing combinations do in this bad state so this is perfect leading into our testing discussion um i do believe that hive is approaching being able to do these types of things and handle multi

client combos you can very i guess i won't say it very easily but you can certainly induce a non-finalized state by taking off you know 50 of the validators and with a bit of extra care you can create some partitions and some building of blocks on different branches like client you've been thinking about this right i have been thinking about the sun and if there's any like really specific scenarios that would be helpful um just let me know offline and we can i can try and lock them out into hive yeah so one thing that obviously comes to

mind is don't finalize and build on at least two branches where the fork depth is 10 the fork that's 100 the fork depth is 200 you know and make sure that you don't destroy the world by doing so um so that's that's one of the avenues other avenues would be there's there's some simulation testing going on with a company called kurtosis there's a desire to do some more live network scenario testing where we dynamically build out you know 100 node network run a test for a day where you create partitions and do non-finality

and things like that so i think hive is probably the hive and maybe kurtosis are the closest to being able to do some of these things but it's certainly critical to get that in there because we need to test all the client combos kind of continuously on that front um i know there's a lot of parallel testing efforts going on right now are there any other things beyond some of the hive scenario testing that we want to discuss today i don't know if this has been brought up

before i'm assuming somewhat yes for example there are certain um orderings specified in the api course that the beacon and the execution clients need to adhere to for example before i'm saying okay before i'm sending a fortress something to a certain hash it is expected that i'm going to send an executed payload with that content and only after that tell you to reorder it however my request or question is what is the

desired behavior in case of protocol violations now yes ideally every beacon chain client every execution client will follow this fact but i think long term it's [Music] it can be expected that certain bugs would occur or certain weak conditions like swapping out an execution client swapping out a perfect client even during runtime and maybe that would result in these orderings to be violated and in that case i think it's kind of important to be able to handle these scenarios plus i think it's important to have some tests for these scenarios

for example if i just announce a portrait update with a hash that's that has never been advertised for anything anywhere right mikko what's the state of handling race conditions between various methods in the engine api is there anything specified there or did we leave it as unspecified i think that um yeah this example if fuckture is updated it will just return syncing because it doesn't know the payload that is specified as the head so i think it's pretty uh it should be specified and dra and all edge cases should be addressed by the current specification i mean so it's it has meaningful

responses and all these kind of cases okay then what we should do though on this pass is as we make any modifications do triple sandy check that um ordering can be handled elegantly yep and yeah and with respect to ordering uh yeah there is like the statement um in the spec um that just says that the fortress updated must be sent in the same order by seo clients

so that's that's basically it so no um no json rpc no json rfc request id is used anymore to order the messages to order the processing of messages yeah you know it's relevant i don't understand that the specs clearly state what the order should be i'm just saying that bugs happen and the clients shouldn't go valley out because somewhere there's a bug thumbs up yeah yeah yeah i get it um i don't think there should be no unrecoverable uh states that the software will be in um like if if the factory is updated and

prior prior to the execute payload if if it happened like accidentally once or twice it will be recovered if it happens always so it will always be in syncing state and yes that's that's the issue so but generally yes it should be there should be no uh like state um from which the software can proceed and recover from with big currents back so intermittent ordering failures should be recoverable but persistent ordering failures you say likely is unrecoverable but but important to maybe induce that

in a hive or other environment yeah but if there is the constant miss all the in the messages i don't see like any potential way for a client to recover itself um any other testing updates and there's a lot of parallel efforts right now or other just testing discussion points for today uh could i just ask what can census

clients support starting from a merged genesis state sorry merge pre-transition or merge post-transition sorry merge pre-transition i guess are there any consensus clients that don't support starting directly at that fork it is specified in such a way that it should be possible i know that in amphora not all clients handled it at that point so we were doing kind of the progressive forking but i'm not sure where it stands today yeah sick i see you unmute and mute um

we can't hear you and i apologize if we've been talking over you for the past 40 minutes still know thank you uh i believe that nimbus was the one and for that the only one that didn't so i would suspect we're in a direction where most can at this point great thanks

okay um other merge related topics consu consuki related topics i know we talked timelines or we talked like the progression of what we want to do with test nets um a bit last week the on the awkward up call we do have spec updates coming uh this the semantics of these two calls are probably the deepest thing um there are a couple of other very minor things that will come out and there is an active discussion and hopefully upcoming pr on um elcl communication authentication so i think probably the semantics of these two calls and the authentication

are going to be the biggest changes that come until that comes out and we get some initial testing going on kintsugi will probably remain up then another version of kintsugi may be named something else then we talk about forking public test nuts so high high priority to get these spec changes out and move towards a frozen or near frozen state so we can get the next wave of stuff done any questions or comments on that okay

great before please i just had a tangential point um does anyone everyone already support the paratrix renaming so could we organize like an update round sometime end of next week perhaps so we can just change the conflict everywhere as well as documentation yeah from what i can tell clients are ready uh in various states and at least have pr's ready so uh we can probably coordinate outside this call to pull the trigger on it on tuesday wednesday something like that okay tell us

yes i have a few questions so it would be really great if somebody could uh collect the information what other clients uh changed and what basically patched and why uh they did so during the this this consul incident as it's a bit not clear what exactly was needed to change in the client so maybe everyone could check this list and see their need to do the same thing in their own client uh well that's my question maybe maybe somebody could do that and another thing so i was thinking that the way

el currently handles forks it's it's been interesting from from proof of stake point of view because it looks like in the proof of work it handles forex well well the minute is running and there is no major issues and i was thinking maybe we can change something and the forking let's say dynamics or economy in order to make on a proof of stake level in order to make it more similar to proof of work so basically the the first idea that came to

me is that in a proof of tickets it appears that the forking itself is is very cheap uh you you can basically if you're a proposer uh you can cheaply without any damage uh let's say build a block from on some old on top of some older blocker and this is something that is is not [Music] easy or not cheap on the proof of work side so so basically the idea is maybe somebody already discussed this maybe it's possible to make less forking or or the the forking dynamics that is more close to the proof of work so that would make

reusing the exact approach that execution layer is using now so we don't make a a problem for execution layer so was there some ideas like this i remember there was some discussion like this um so uh i'm not sure if it i think it was in in greece where we had the discussion whether whether proposing bad blocks would be uh would be an offense so yes some penalty or or even

maybe not slashing as strictly with slashing but but at least some some large penalty for uh for making something which is uh unusual in proof of work [Music] economy and and which is very uh let's say tricky for execution layer to handle right so defining these cases one is hard and two um might induce arbitrary penalties because i happen to be slightly out of sync or uh on some fork and i uh haven't been able to resolve it so the right there's but that's the same on on the proof of

work if you are late if if you are late on the on the chain and you don't follow it and you build the spend your resources uh building a block that is not on the latest block then you are penalized by wasting resources right i understand i mean yeah i mean i i understand that this is very large uh very high level uh idea and of course it it's it's not complete without all the details but i was just wondering maybe there was discussion right essentially if you can include proof that you that someone built a block that did not make it into the canonical chain uh you could

potentially penalize for that to try to induce the forking costs that you would see exactly yeah but but you can do that this is i think it is possible you just have this blocker which which is signed by proposer and it's it's it's it is pointing to some old block so you likely can do that uh because there is a block roots accumulator and so you can take the header of the proposed block and you can point to the proof of what was actually in that slot in the canonical chain and you

could potentially decrement the value there i think one thing that is an attempt to do this without having to directly incentivize that would be the fork choice being an attestation-based fork choice such that it's very it should be moderately cheap to figure out if it's even valuable in processing something especially with respect to the execution layer um but yeah i it's an interesting explanation that's not something i thought they believe though okay yeah if somebody wants to discuss further yeah we can

maybe do that yeah mario's with respect to not following the consensus rules i think that's much more difficult to actually improve on chain without more maybe advanced snark constructions um about the beacon chain execution which arbitrary proofs about beacon chain execution are certainly not feasible today all right

let us move on um are there any other client updates other than what's generally been going on with the merge that uh consensus layer clients like to share today i would like to welcome daddy who just joined the team so yeah welcome daddy welcome to the team also new release three zero three three zero and we're gonna drop the user guides hoping to contribute to client diversity uh yeah that's it for us anyone else

yes so for grandina we finally joined the testnet just before the crash and we thought that maybe we caused it but that that was not us and uh and there we are back again today with 1000 of our data risks and looks like it's performing well currently only we test it only with gas and we will need to test with other combinations and maybe somebody could comment is if if if the implementation is working well with death is it likely that it will just work well with others or most of them needs

custom handling the short answer is we do find idiosyncrasies and are attempting to work through them and test them such that in a way that we they don't appear uh by accident but generally yes uh if it works the gap that should work with the others okay um i also think we're we're at a point where it's uh it's pretty stable so in the beginning it was uh more of a hassle to get uh like the different cl's with the different yields running but now

we seem to like converge on the common implementation good any other updates yep i guess i can join that so on the prison front we have been working on optimising sync we started looking into four choice proposal boost and the new for choice back test hoping to get those done by the end of january and um we are also switching our bitcoin state from um grpc protobuf to a native uh

global structure so that will save our memory and we're also implementing the new web 3 center key manager api so the goal is to support a generic gui coming from the coming from the 8th stage effort we also proposed a few more api a few more key manager api points to manage public keys better and we're on track to fully support um web designer um api by the end of january and we also hire one more person tehran and he is our devop guy so if you see him say hi awesome thanks terence

right now provide some updates from nimbus we are also preparing a new release which is probably gonna be shipped today or tomorrow it's mainly a performance release but it also ships the key management p major api that transmission uh interesting thing about in this release is the fact that with through optimizing the use of the nimr garbage collector a little bit we were able to significantly reduce our memory usage and so we are now hovering around one gigabyte on mainnet and we are pretty confident that our next release will kind of provide stable operation always win less than one gigabyte

that's insane congratulations uh although that's the interest other interesting feature in this release is that we've implemented pretty much compatible validator performance metrics which are very similar to the ones in lighthouse i would say almost 100 compatible i've noticed that prism has implemented validated performance metrics on their own so i'm curious whether there's some appetite for standardization here so the dashboards of the users can be portable and the other line of development that we want to pursue very actively is the light client with great i think that was mentioned in

the past we have a server that is compatible with load star and now we are trying to build a client but within nimbus there will be a special lifeline mode in nimbus that enables some of the uh some of the rest apis remain active while others which are not possible are disabled but to this end we really want to see the light and updates being distributed within the peer-to-peer network so we would try to propose i mean to create an implementation that works for nimbus nodes and to try to propose this as a spec thank

you else uh yeah i can do lighthouse oh sorry you you go um agent all right thanks um so for taku a very quick one um we've got a 22.1.0 release out which includes the network config for kintsugi so you can just say network consugi and off it goes um uh also a bunch of optimizations in there we've been doing a bunch of work on optimistic sync and finding corner cases stressing guess that get out by by pushing some of the limits of what we validate and scent with

and generally getting into that kenzugi mess um that's been really helpful though so i think we're we're starting to find lots of great corner cases in optimus to think and i'm feeling a lot better that that it's it's not only viable but getting well understood and um should work out pretty well uh we've also been doing a bunch of work on the key manager api um which i think james he was going to talk about a bit but um we've got that behind a dev flag so you can enable it now in the release build um and play with it uh the bits we're missing before we enable that are just making sure that authentication is

supported and ssl um but that's that's underway now so hopefully in the next release it should be a properly supported feature to have that the key manager stuff there excellent thank you and paul ready so we had a nice break over new year's it was good to get um a little bit more relaxing break than last year when the beacon change was launched um so we're gonna do a release candidate in the next 24 hours and hopefully release next week it's a big release actually a lot of work there's some improvements to attestation performance also networking and we removed some cases where we were down scoring piers for our own

internal errors we're also moving our slasher database from lmdb to mdbx because it supports pruning and shrinking the database that's uh so we can save a lot of your space for users there i've been working on optimistic sync specification feels like it's getting pretty close to being mergeable it seems like the i guess the the core concepts of it seem pretty stable which is nice i'm going to start working on implementing that in its entirety next week we worked on a flash box proof of concept last year we made good progress with that got something working but we haven't been playing with it this year yet michael did some more client diversity analysis he's also looking at doing a little bit more

in that department and we are ready with proposed boost and also the bellatrix rename um so happy to uh coordinate on those offline that's it for me great and it looks like marius had a question for you paul harris well yeah it's it's more for like all the other client teams because right now uh during kinsuki the the chain databases grew extremely by like seven gigabyte bytes per day or something um and because of non-finalization and i expect that once finalization hit then

the the uh the notes would would automatically prune these old states away but they didn't so i was i was just asking if like people have already looked into this if this is um uh if this is being worked on uh or if some clients already have it i i didn't test all the clients i only tested lighthouse but it seems to me that like that's that's pretty like a big price to pay and if we expect yeah we we we should prune and shrink it we'll just have to look at why it's not happening it's we've done it before we know it it shrinks so there must be something going on all right that's that's great

also for such it seems like something like on the order of uh one state per epoch one full stay pretty epoch's being written to disc in that case um are there any investigations to do de-duplication of states when non-finality in writing or potentially in light quote archive modes so just like handling diffs because a lot of the validator set doesn't change i don't know if this is like a high value item but i'm just curious if anyone's doing that deku's new uh taker's default mode for storing finalized states now stores them as a tree so it deduplicates um which gives you really nice fast access to historical api queries nice um but we still store the full states as

snapshots for non-finalized stuff because pruning is really hard once you start storing a whole tree of things and have forks involved i see i see okay uh is this working though yes welcome thank you uh yeah i mean thinking with numbers we actually store everything no we store the validator database deduplicated also for recent states because that one actually doesn't change between forks so we shouldn't be growing very fast and we've we have that nice fast historical access as well cool cool so

so when you do deduplication and you do diffs on historic stuff that means that you can do a quote archive node at a much lower cost yeah exactly cool like nimbus doesn't even have a non-archived mode right now so nice you always get the full history it's just sitting there and it's not that expensive yet it's gonna become more expensive but the point is that the validator db trick is very nice because the validators are the same across all forks so you just it's kind of like an append only list

right and so i forgot to mention actually one of the features of our latest release is a cache in the rest api that speeds up what i call historic traversal traversals when you request certain information such as the validated balances either slot by slot or epoch by epoch this cache in the rest api speeds this significantly so that's one of the differences with the way teku does it we've deliberately used more disk space um and we're storing every state um deduplicated but in full so there's no there's no replay or calculation of

those dips it's just there's a tree there and you load it we can load partial trees there every single different epoch every block effectively okay well um so it's 200 gig i think or so so it's it's pretty chunky in terms of what it stores for the entirety of mainnet but you query anything instantly which is quite nice all right and this this 200 gigabytes is for for this state stream or there is something more that that's the whole database so it's got every every node of every state from each block

right through history plus all the blocks plus whatever other little bits and pieces we need to store um so that'll be east one data and so on but do you know what what what the percentage or or what what the number of gigabytes is for the state 3 only no i don't and ben's just corrected my numbers apparently we're using about 350 gig i'm terrible with numbers so you never trust me um okay it looks a bit a bit much i would say i don't know i mean it would be great if you you if you get a more accurate number of uh how much is for the state realm

that would be interesting to hear yeah we store everything except the finalized states in under 20 gig 20 okay like it's it's quite small um so pretty much the entirety of that is is all the state trees um but it's designed particularly for people like in bureau whether you're having quite random access to states from all over the chain and quite a lot and you're prepared to throw some disk space at getting it quick um i mean it's okay but but this states uh state three alone it probably i mean there are not that

much of diffs uh really i'm just thinking you know it should be if you don't uh if you have like a full duplication and you don't uh store anything extra i don't know like maybe you calculate some rewards let's say for each state you uh you cash some data rewards or something then this is extra data but the the state diffs i think they are pretty small and and if you if you just have state tree uh which is fully duplicated it should be really small not a hundred of gigabytes or so um

yeah i mean there's there's a lot that that changes in there so it depends how you store it the actual data that changes is quite small but to then track which part of the tree that data changed in and kind of a patch format you wind up potentially having quite a lot of overhead if you do completely full deduplication so we store it as a merkle tree um it's very kind of old-school eth1 um and and wherever the the nodes are the same we don't need to store it again but we can we can walk that merkle tree down and load just a part of the state so if you ask for balance of one validator we walk the mobile tree down we've optimized that to reduce overhead by being able to compress

and store you know five layers of the merkle tree uh in one go so then we only have to look it up once it saves us some disk reads at the cost of not deduplicating as much and so we found that five number by doing a number of experiments and that actually gave us the smallest disk space um if you totally deduplicate you wind up with more overhead as well and so you wind up using even more displays okay clear because the the the the key thing is that like in grandina we have very fast taste transition and we started to think is it is it really worth to to optimize that much on the duplication uh when we can

transit the inner states pretty fast you know and uh yeah it's it looks like it's it will be always a trade-off between between the disk space anyway and between computation speed that would be interesting if somebody if teams could provide their you know numbers like how much uh the statory uh takes on a disc and and what speed ups actual speedups it brings that would be interesting to see as it looks uh that the degree of the duplication or the extra data is different um

in different implementations it's uh it's not the same on all the coins it looks so that has this uh the duplication stuff implemented okay i think we shall okay one more point i could share on this topic which is that uh we've been uh playing around with this idea called era files um so an era is basically a dozen blocks of data why a thousand because that's the number that appears in the number of

roots tables in the state and it's about a day of data so if you take mainnet up until now we've had about 340 such eras and if you count the amount of space that that takes so in each era file what we do is we put all the blocks all those 8000 blocks and one full dump of the state so basically from any era file you can just you know start from that point in time as a say a checkpoint sync or whatever um if you take the full history of ethereum to date

in this era file format which is just a flat file with blocks and then that one state and you end up with a with about 30 gigabytes of data today and that's just raw no indices no no smart things nice and i do think it makes sense to get those changes into that next fork that will be coupled with shanghai oh you mean the block routes yeah that would be nice um but this is separate actually this is

just like um right you don't actually need the change in there not for the year well for the era files it becomes a lot easier but this is kind of like the nice thing about this format is that we can generate like an error file every day and it's like a fully verifiable format against any later state more or less and you can distribute them through torrents or whatever you can share them with your friends you can serve get blocked by range requests for from them they're just nice and easy to work with and yeah the block range thing would definitely make them even easier to work

with like for lots of reasons cool it's interesting to see um implementations what they've been optimizing and kind of the end user they're catering to diverge more and more i think that's a good thing um okay anything else today any uh research updates any general discussion points yeah maybe

an update from my site many of you probably saw it that i um that i published as a sort of a draft for some changes to the sharding that should greatly simplify the design um so i don't know which teams are currently already uh or have done work on implementing the previous charting spec um if you are on that then it might be worthwhile pausing that for the moment and having a look into the current draft pr that's up it's a very early stage but it would still be interesting to get feedback and to get people to look at it at this point and see what they think of it

i think it's a great simplification will make it much faster to implement yeah awesome if you're listening to this call and you're networking wiz and wanna uh work on some fun dht stuff get in touch with us also if you know something about dhts then that could be very helpful for us because i think most of us don't have much experience anything else

i did see george on our team dropped a uh pretty comprehensive ssle secret single single secret leader election proposal on eighth today if you want to check that out this should be some good content and if there's nothing else we're gonna close going once going twice okay thank you um like i said hi hi high priority to get any spec changes in asap so we can keep the merge moving

talk to y'all soon thank you thank you thank you [Music] so [Music] [Music]

[Music] [Music] you