Holesky Debrief

Transcript

so I I won't recap what I just said if that's okay Tim no I'll good yeah um is that enough context for everyone basically the initial priority that we're looking at is trying to get back to a state where we've got more blocks in an Epoch than missing because the priority is to get to a state where we're relatively healthy if not finalizing because to get to that state then we can um so I'd like a Miss slot rate of more like 25% or less um yeah I was just curious where we're at right now we're at we a few epochs ago when I wrote a message in

interrupt we were looking at um over the course of four epochs the worst was four in an Epoch and the best was eight in an Epoch so we're at about 25% best case coverage at the moment uh and we would like that to be more like 75% before we start issuing a lot of slashings um and the reason for that is because slashings are going to cause a little bit of chaos around the place um and they're inevitable yes but we need to get to a healthy State before we start destabilizing things hopefully that's enough context

for anyone but I'm happy to answer more questions about why liveness is more important than finalization if that's helpful maybe one question is why do what's the trade-off between them would explain that for non devs like why can't we just do both yeah sure both is a good idea um finalization is going to be a long road livess is a short road so um livess is the prim concern but also to get back to finalization what we need to have is 2/3 of our effective balance voting on the right chain and to get to that point um it would be nice to First

understand how many validators are on the canonical chain and proposing blocks because that will give us a better idea of how many need to be slashed and so while we could all issue Mass slashings uh we could also potentially be able to distribute the slashings and have less per team and therefore everyone hopefully has some valades left at the end of this event got it um so glio the the whole point is we can't stop all the slashings but we can stop some slashings so um

basically if we get back to a point where two-thirds of voting then whoever's still in that surround vote situation and avoid voting and we still finalize if that makes sense we haven't lost liveness we jeopardized livess if so if no nodes can synchronize to the chain um then eventually we do lose livess but liveness can be maintained as long as we've got enough nodes following along in what way do you think we lost livess SL C we weren't making any blocks the the honest validators could not find the

honest chain that's kind of the definition right well that's why we were shutting nodes down yeah so we this is the problem right if we've got so many nodes that are on the wrong chain and it's hard to get to the right chain then we do lose leness and that's what we've been trying to avoid and why we've been trying to get more blocks because the more weight we add to that chain the less likely we are to actually completely destroy this thing right but we should be robust in those types of scenarios like we're not talking about 5% or less of validators that were on an honest chain it should have been 20 25% and that's an amount that they should have they should have found each other and they should have continued building a chain albeit like you know very

sparse so there Sy issues but the core issue and and the start of this whole problem was that a super majority of nodes were choosing to verify that a that an invalid block was okay on the Chain right and then things started building that's where this problem started now how we avoid that in the future there's going to be opinions that we can look into later um probably not the form to dig into yeah I think we have time to do like a deeper retrospective on this but I agree that like um at least like on the sync side uh like yeah there's probably some stuff that should be improved um yeah yeah um

ran Julio do you have your hand up to comment on that uh yes so one of the goals that all uh B here is well improving the Miss slot rate and while we were ready clear in in the so heski St right now is at roughly 5050 split between Cavs and the community and while Cavs are more or less knowledable on how on what to do right now uh even though we should formalize the instructions we still should give clear instructions to the community because we'll never get to

our Target of 25% Miss slots yes and I know yesterday uh on the interupt Channel people were saying they'd want to make sure they can reliably like sync their like client teams can reliably sync nodes and add them to the to the network and whatnot um before we sort of instruct the community to do this but do we feel like we're in a spot we're now putting the message out uh to in like a clearer way to everyone would be helpful we're close to that but we've only really constructed like two nodes I

think successfully so to be able to give it to people to to replicate I'd want to be able to do it on more than a couple of our notes because otherwise it's just going to be in support that okay uh I know that a couple of people in the community who have like decent amount of validators already uh have their notes synced but they just they're not enabling the validators should we tell them to do that so my message Remains the exactly the same as on the interrupt Channel please leave your slashing database your

slashing protection database in place but please enable your validator that's where we need to be we need to have the validators active and the slashing protection will stop you from attesting which is fine if if you're in a position where you can attest with your slashing protection that's great that's actually even better and there are are a percentage that won't have upgraded so they're going to be in that position okay so run your validator if you're synced keep slashing protection on yes and then the next step after this will be um a controlled of slashing we don't want to go crazy excessive because

in the past we have seen slashing events that are larger um start taking noes down so we want to kind of dip our toe in the water and get a rate slashed but we don't want to do crazy amounts got it and if people do not have if their not is out of syn should they try to get it synced up their head yes so for any Cent that have put out releases already I think those e clients some CLS have and we probably will tomorrow these give you a good chance of getting back into sync um some people have got databases floating around uh I won't speak to each client's position but we may end up having a

database floating around as well because it seems to be a lot easier at this point with the forks okay um quick question if this was a like like a real scenario at this point we would like I don't know someone we have like I don't know 30% of the network is is uh is uh is going to get slashed and they know that they're going to get slashed either now or at some point wouldn't it make sense for them to uh to exit uh to to get slashed as quickly as possible so they they can

get their money and and and move it and if if that is like the rational choice for for validator that is going to get slashed anyway um this would like like massively increase the problem for us because if we cannot deal with these slashing events and the rational choice for validator is to get slashed as quick as possible um yeah then that's kind of bad right yeah the way oh go ahead uh I was just going to say you are right in in that the first people that

get slashed that get their slashings into a block are less affected but everyone's going to be affected in this event so um you're going to be active but slashed on the chain for a period of time before you get to exit and the rate of exit is gated so we can only slash we can only exit eight validators per Epoch this is going to mean that of the order of we're probably talking a couple of hundred thousand several hundred thousand valers it's going to take a couple of weeks to exit them all um and for that reason it's going to take a couple of weeks before we get back to finalization so finalization as a Target is a good goal but it's going to take a

couple weeks I think um M's numbers we're around 18 days but it depends on how many valid is exactly we need slashed okay I want to keep us on track to fixing heski we only have 45 minutes left uh there'll be a lot of like long-term things we need to figure out coming out of this yeah um so um to recap like we have uh you know part of the chain uh part of the chain live um we want to grow that we want people to bring up nodes online ideally minimize the number of slashings um then what do we expect the next steps

to be um do we just gradually keep iterating on this or yeah I think I think we want to do a small number of slashings at a time initially and the reason for that is just to make sure that clients stay up sorry just to yeah um for example well it's pretty easy you just remove slashing protection database um from a certain number of keys yeah but how do you do is low it's kind of right okay no no for for example we can only exit 20 we can only exit eight Keys per Epoch if we were to in one Epoch enable

remove the slashing protection for say a th Keys then they're going to take a number of box to exit we're not actually in a hurry to get these slashings to happen we just want to make sure that the nodes stay up as much as possible because the more nodes that can help maintain livess the more likely we are to get back to finalization I was just thinking about the logistic of the 1,000 keys at a time I don't know the coordination looks onesy honestly so that was just my concern so so my my initial hope is that we get enough validators online so that we know the depth of our problem if we're talking about needing to get 35% of the

validators on the network slashed then there's enough people in this room to do that and we can divy up that amongst the teams and say 10,000 here 10,000 there and we can say you go first you go second and coordinate this somewhat if it was on Main net and the large slashing happened they're all on the network and it would be interesting m okay and what about the timings because right now some valid I mean most of the validators are currently live uh are all all of the operators at least are all getting slashed and since nethermind already lost 10% of their validator to slashing so sure so if you've removed your slashing database which is um not

ideal then you could be getting slash right now yes yeah but it seems the majority I mean seem a test um nether mind and some others are getting on also light is getting slashed so yeah the thing is that if if they've chosen to remove their slashing protection they will get slashed they they have performed a slash action yeah okay what I'm saying is that right now is that right now most of the blocks are produced by these people and if they just get and if we don't act in a timely manner the chain won't progress anymore because they this is why we need liveness right this is why we're pushing for liveness the more valid is the Run line that haven't removed their slashing

protection the more asking about the timings I'm that's why I'm just asking about the timings here are we need to get some information before we go to gangho on timings so we want to get as many blocks produced as possible and then get a small number slashed and see if we have problems across the network and if we don't we can increase the rate okay and we've seen slashing being produced some slashing been produced already I think um and do you have a do you know what the rough number has been or rate has been I haven't looked to be honest I've been concentrating on getting no most L 10% you can check it on D okay and so um okay so we want to get blocks coming reliably then we want

to slowly scale how many uh slashin we see um what's the mechanism to coordinate that I think we would start with the alisy Genesis validators set um and the CVS and number of CV teams have like 100,000 validators so it's pretty easy for us to coordinate between those groups unless anyone has a better idea I think that's yeah um so once we know a reasonable number we can determine how many each team's pretty much going to have to lose

and we can probably absorb most of it within the CV group um we may have to get some SL some enemies to get slashed a bit as well yeah okay so uh once we do have or in parallel to this um Amic has a message and we should have a list of the holis ski entities and their is I think was it Roman who put together a a chart yesterday with like the attestation rate um oh do you want to maybe give context on the message you just posted oh yeah these are all of the organizations which

are not cor devs who have a decent amount of validators and and they're part of hes Genesis pool okay and though and we again we don't have statuses for most of them uh okay so uh I copied this message uh from the one I sent early in the morning and some of them were able to provide the status back in the telegram chat but I don't have this compiled I just send the the list got it and so my sense is there's like 1.7 million validators on Hest we have like 12 cent teams and there's like ballpark uh 300

400k validators there so like it seems like we're in touch or like you know the vast majority of the stake is like in aware of this and in touch is that a correct assumption oh yeah Terrence I just want to mention that like by definition not all the validators work as slash right you're only uh in danger of getting slash if you voted on the invalid Justified checkpoint but it's not clear to me who did that who didn't do that so when we plan this we should look at who are the validators that voted for the invalid Justified checkpoint and then based on that number we account for how do we uh

Slash them yeah is anyone looking into this yeah I think it's off the order of about 40% maybe 30% but I'm not 100% sure bearing in mind we'll also get this information on chain because when a validator comes back online if it didn't attest uh to the incorrect chain then it's going to be able to attest normally on this chain the slashing protection won't stop it from doing so so we'll start to be a to get numbers from chain as the validators come back online so pretty much self selecting right every everyone who managed to bring their validator back online we'll we'll see those numbers flow through the chain okay so but and just to make sure sorry I understand as an El person if

you bring your validator back online with the new release and you don't mess with your NT slashing database and you attest this means you didn't attest to the wrong finalized chain correct yeah that's exactly right okay okay so I guess then like yeah the instruction of telling people to come back online is still probably the main path forward and then of these validators that come back online who are not ATT testing uh we assume those are the slash ones or at least highly correlated with the slash ones and then with those we can coordinate a response does that make sense yeah so at that point we know the Gap and at that point we can say you know we've got a deficit of this many to

make 66% happen Okay the the other thing worth mentioning briefly is obviously any validator that's not online and in fact every validator because of testing will be going through inactivity leak as well so they'll eventually get get kicked out but that's a slightly different system and the balance isn't zeroed straight away and a whole bunch of other things but you know again if if people don't come back online we're not kind of stuck in limbo um if the chain doesn't finalize because enough people are either slashed or not online then you know eventually they'll get kicked out anyway by the system but again it's a relatively slow process because of the execute got it and the other active question that I have for cordev for consensus layer is

what happens if we have an active validator that reaches zero balance on chain because I don't know what happens to that validator in that instance and I don't think we have any tests for that but that's just an active question to think about unless someone knows the answer off the zero once they get a 16 effective balance they're exited right no they're put in the exit Q so if we have 400,000 validators in the exq then a lot of those reach zero okay so it's just kind of question if you've got an exiting validator that's at zero if I've got an active validator that's in any state that reaches zero balance does it just become exited or does it cause some sort of Gap in our spec because that's going to happen I

think through this because at the rate we're exiting a lot are going to reach zero so it'll be an interesting experience and there's no tests yeah there's no tests for like zero balances well it's a weird case because you should get exited at 16 16 but because of the situation we're in um it's really into and I put a call out on to it in a couple of channels um daan's um Devcon talk which goes into craziness that happens once we get a mass slushing event got

um okay uh Jim you also have like yeah some points around mitigating uh like yeah mitigations we can apply given this issue I think I would pause any like long-term conversations for now and F focus on how do we actually get the chain back online um it does seem like we have a rough plan uh does anyone feel like yeah there is something broken or wrong here or is everyone on board with this approach of okay let's get those back online get the ones that are upgraded and hav't

attested to attested the the right uh minority chain um get a of the one um of the ones that have attested to the to the um majority chain and then cordinate slashings does that make sense to everyone yeah Roman oh one question to make it explicit we will consider the chain quote unquote live or Lively and we'll start considering uh the s when we have 25% missing slot rate is that the number I'd like to be in that area um

the less Mist slots the better because once we start injecting a lot of slashings we'll see pretty easily that clients are having problems if that rate increases a lot okay so yeah summarize that get nodes back online um the the ones that didn't attest get to attest to the the correct change the one that did attest we just keep on hold for now ideally bring the number of Miss slots back to 25% or less then start coordinating the slash chains um I guess one question there is if we quickly learn that uh like uh I guess for uh we're going to

have to to leak or or or slash a bunch of validators before we get this Miss slot down to 25% so like do we is that is that like a correct assumption like we're going to be probably waiting like several days if not we can produce blocks straight away with those validators so the there's two separate concerns there's attestations which are the votes that we're talking about that we need to get twoth thirds of and there's block production so we can produce a block safely without getting slashed if in this scenario where we're in now wait so a valid what I meant is a validator who's previously attested to the finalized majority chain they can no longer produce a block on the Val yeah

they can yep well without getting okay well so we that means we should okay yeah block slashing you're only pretty much if you're signing two different blocks on the same slot so as long as you're going ahead and you're not going back with your block production you're fine to produce the next block okay and and this means that if I'm running a validator with like the upgraded updated release um that had previously tested on the wrong chain then it's still valuable for me to get back online and try to produce as many blocks as possible absolutely yep so the 20,000 keys that we've got on our teu node are producing blocks now but

they're all slash for ad a station once we change the database over okay um oh that's wild I didn't know that um and then and and then yeah and then so and then these validators don't need any like special setup basically like as long as as long as they have their slashing protection on they're not going to test and so they they probably won't get slash but they'll be able to produce blocks 100% great awesome um okay so this means that like yeah within hours to days we should be able to get that mid slot rate back to less than 2 5% um it only it like the blocker there is getting all of these operators

actually upgrade their software and start their notes again and uh get up the the head when they sync um yeah on the syncing side so I know this was pretty problematic uh over the past day or so has things like improved is the I know people were using like the panda Ops snapshots um yeah a lot yeah even since this morning like I've been sinking another node this evening and it's it's going a lot better than the one that I was trying this morning is so as more and more good nodes come online it should be less and less of a problem to sync yeah so should people just like try

vanilla syncing now or is there any special configurations they should be running or like should they download a e Snapshot from Panda Ops um I think the snapshots aren't a bad idea if you can if you have a snapshot that you can use with your clients um definitely use the updated clients um there's been there's some cases where we've had to like just service restart um the CL because it gets marked as a bad Pier so it doesn't sync very well so there's little nuances but I think people can try it out as long as they're on the right client versions and just

to make sure so like that people have the the right snapshots like what is um how can people verify that their snapshot is actually correct for the chain we want them to attest to or to sync to the database uh I think the only Dora that's working really well partic particularly at the moment is PKS um it's hard without a node so that that's why I've been trying to keep up with interrupt as well the channel because it's a lot easier to ask someone that's got a working node if you're up

if you're running okay and luckily I've got a bunch of noes to look at but not all people are going to be in that position okay okay so so I'm not sure what the answer is there in a nice way honestly yeah fair enough so if you're if you're like plugging in snapshots into your not then maybe check out interrupt and make sure you're doing the right thing yeah there's a specific we could say a specific slot that you need to be on which is after the bad block and if that slot if you go the wrong direction at a specific point we know you're going to end up in the wrong place and that's actually fairly early in the sync so we should probably Define that really well and make sure people understand that oh actually like yeah Sam uh from pandal Ops has a script that was show in

the chat that seems to do just this which is great amazing so yeah this seems like a rough backward Raphael um I just had an idea maybe trying to improve this uh the sying so we do have like two out of four nodes on our side are like sync um I was considering just taking you know a dayto de Snapshot from the El NCL there and just spin up another I don't know 50 100 notes with that uh just in hope that we could help like on

the syncing what do you guys think is there reason the more good the better okay won't it not on board new Stak till the Chun finalized yeah so but we can have nonv validating noes that still help the health of the network yeah and the main purpose is they'll just help everyone else think better basically yep that's their only purpose yeah okay so we will try to bring up more not yeah even if it's a handful it would be still

helpful yeah and I guess if anyone else uh on this call is listening and has the ability to uh to rapidly do this then that would help at the network stabilize uh so yeah uh pet to Pier is offering to spin up another 20 um do we need a fallback in case the recovery fails what do you mean by recovery fails be done uh like if we um if we cannot like if this plan that we have doesn't work out what do we do I mean doesn't work out

how like if not enough uh peers or validators uh started testing to the correct chain we don't really need a testing we need block production but um basically it's probably not the right florum to go into doomsday scenarios maybe the chain crashes but it is a test chain and we could spin another one up worst case I guess um I probably I don't want to go too far into that because I think it might railroad the meeting um the the other thing I would say is that this specific scenario uh is something we have fixed

this poia so we really want to make sure that the polier Els are up to so that we don't cause a problem there but I don't think it's a reason to slow poers M um for down but that's just my opinion we can go into that tomorrow on E if you want on ACD Matthew so the one thing that's unclear to me is that I understand that we're going to have to start a testing and thus slashings but is the goal in order to reach quarum in order to finalize or are we slashing in order reduce the amount of effective balance in order to lower

the amount necessary for finalization in subsequent epochs and there are a couple of second order effects that I think we should discuss I don't know if this meeting is the right one uh but once you exit a certain number of validators now how do you a test for the second Epoch uh when you're going to have a very long queue for exits and I think yeah like just discussing that process more than like just get State chain stabilized and just get to the point where we're getting to slashings I think we need to really think about what are steps two and three because it will aform the process that we go through in order to

to schedule and Stage those slashings such that the the end goal of of finalization is actually achievable I mean the end goal is hopefully to keep the chain alive and finalizing um the the reality of the matter is if we get of the order of 400,000 slashings that a lot of things are going to go to zero and it's going to take probably a couple of weeks to get to a point of finalization um we need to get to a point where we're stable enough to start that process and we need to keep the exits going at about a rate of eight per Epoch otherwise it's going to take longer and longer

so that's why I guess everyone's so determined for stability first then let's start the slashing going but in reality it's going to take several weeks uh Solus yes so so one thing we we didn't discuss so far is that we didn't uh test uh pectra successfully on a large Network and it was uh and we we only had holeski for that and holeski is not in the shape right now and it will not be in the shape for quite a long time uh I mean like main size network with with a

healthy million of validators so I think it's it's very important to discuss how we're going to to solve that because we don't likely we don't want to deal the uh the pectra relase on the minute or very longer so so one of the ways to to do that I think we could try to yeah to Shadow Fork the holeski uh and in some sense it could be another disaster recovery for um uh let's say for for even bigger disaster than we had right now so for example if just imagine the station where we we

kind of come to conclusion where uh we see that it's not possible to survive the uh some fatality in in at a consensus layer and then we need to somehow save the E State uh in some way for example Shadow for uh or or essentially forking the seal state that was before the uh before the failure so I think we could try to focus on on making U some kind of Shadow Fork on hes which would be based on the validator said that was before the um crash right so I guess yeah I want to

make sure we fix leeski first but I agree that uh yeah having a proper pectra testing on a large validator validator set to be valuable um would so so is the thing that's required to do this Shadow Fork basically just a snapchot of holis el and cl State prefork and then we would Shadow Fork from there so uh I don't understand this suggestion because for that you would need every uh Genesis validator and every hki validator to basically spin up

a duplicate setup to participate in this correct and then to the extent that we have uh a super majority from cevs like maybe that's possible but we don't have a super majority from Cavs because it's a 50/50 stake split um and yeah we have done Shadow Forks before without that but usually the shadow Forks just test the transition and they kind of degrade pretty quickly afterward so the other option is yeah assuming we have like 50% of the state like yeah if we don't have enough to finalize is there a valuable test that we get out of this yeah but we can bring maybe part of the

of the community to make make a bit more than two3 of the state I I personally think that like we're stretching the community enough involving them so closely in Hy recovery efforts and we cannot expect them to participate in this experiment as well yeah but from what I remember the heski Genesis Keys a lot of them were given out by eth Pander Ops so they may well have the private keys for most of the Genesis validators already so they can just pick up and run those those validators themselves we don't actually need the community to step in and run them oh we don't have the keys for HQ

validators it was another process when we set them up um every team basically sent us a list of pup Keys instead of handing with theonics I goes that idea yeah okay uh Alex you had your hand up yeah I had two things uh the first one uh let's just well I guess I wanted to confirm so we wanted to wait for a particular number of filled slots on the chain and I guess we said 75% do we want a Time cut off like let's say you know we get to like 4 or 50 in like the next six hours what if it takes more than a day to get to 75

do we want to wait for 75 or have a cut off I think your potential for things to go spectacularly interesting directions is very high if your block production is quite low so uh sure we can try it but we have to accept that this test net may completely fail if we do enough bad things to this nowor right but what's the concern just that there aren enough blocks and then we spam it with slashings and then everything falls over we spam it with slashings enough clients have enough

issues that we don't have loudness anymore and we get to the point where we can no longer sink that's how Gorly kind of went we didn't spam it with slashings but we ended up with not enough validators that could maintain the state of the network and we lost liveness before we wanted to shut it down um honestly if we just want to shut this down anyway then we just pull the pin on move on with our lives that's an option and it probably saves us a lot of hours but I think we should try to recover so that we know how to cope with these things properly yeah I think it's worth trying to recover because we're heading in the right direction so then do we just want to see how it goes over the next say 24 hours and I guess we can make a judgment call if it doesn't look

great that would be my suggestion get back to try and get back to as many client like as many get documentation out to tell people how to to sync their nodes and try to get as many valid keys back up producing blocks and if we can't do that then sure we just start seeing what happens when we slash things but yeah okay okay yeah we can make add a second thing as well yeah perfect um well Tim go ahead if you're just GNA make a comment about ACD I was just gonna say like yeah let's make a call about this on Al cevs tomorrow like we'll see to and you know if things are

going well even before that then great but if we see that we're like at you know 55% on aord devs tomorrow and out of ideas about how to get that number hired and um yeah we can decide if we want to start triggering slashing yeah okay uh and the second thing was Mario had a nice question here so I believe if people have slasham protection setup correctly they could just exit on the good chain um and we don't need to like have a mass slashing I don't know if people feel like one route is favorable over another yes but also you can't exit while we're in Long inactivity so uh long non-f finalization

so we like something has to happen to get back to the balance we can't just exit validators while we're not finalizing I could be wrong if someone wants to correct me but I believe that validators can't exit until we're finalizing I'll go double check but if so then yeah the slashing is the only route then so reduce the balance um Jim yeah just just wanted to say I know this were focused on heski here um we probably do want to talk about MTI for

this event before shapoo and also if we're going with shapoo now you know if if if the time for that's tomorrow that's fine um but obviously that's uh you know only only the execution air guys are on there theoretically tomorrow and the conversation is probably a little bit wider in that you probably won't buy in from consensus layer as well if you know for what happens with saboia yeah I guess maybe to address the E Point like every CL team please send someone tomorrow uh it's important I understand some of you are in uh Australia so if not everyone on your team can make it that's fine but like yes if there is not a rep from every CL team tomorrow that will be bad so please

have someone there I think yeah so for sepolia one like the main concern is do we want to put out a release that delays the fork or removes the fork um and get everybody to update ASAP on the El side people have already put out releases that fix the issue so um and I've been waiting until tomorrow to update kind of the announcement and potentially make another announcement because uh what we may decide but the risk of delaying sepolia is we now need to get all the validators to upgrade grade to an yell another release um and then a subset

might actually go through with the fork uh it's a fairly controlled validator set so it's it's maybe not that complicated um and yeah what I would say is is it generally speaking and again we're seeing this with with Paul and semolino that the general feeling so far has been not to delay of polio so I guess the real question is can someone come up with a good reason as to why we should delay this a polio because obviously you know if if if there's a good reason to delay it that's fine um and we kind of want to think what that may be so if anyone out there on this channel or listening elsewhere can think of you know a good reason for us to delay that that would make sense but I don't think we've really bumped into a good reason so far given we understand what failed

on heski and we believe we've we've addressed that and and again we'll do it tomorrow there there's more mitigations we can talk about but um that that's probably the key question you know can is is there a good reason if anyone can come up with a good reason then it would be really good to hear that sooner rather than later I haven't heard one either and then the implication if we don't theia is that gu uh Basu and nethermind users need to upgrade their software that's correct um Okay so Okay and like yeah I think we should

make that final call tomorrow on the call but it does seem to me like yeah there's not really a strong reason to delay it um I will update the blog post today with the releases for nethermind G and Aragon um and then yeah if we choose to make some more drastic change tomorrow that affects every single client release uh then we can um can make that decision then um there's a question I uh sorry I'm bosted the chat but uh saying we can't test consolidations on sepolia is there a reason for that I think it's just back to being a perion validated set right you know yeah so in inoo you can only become a

validator if you've got a token um I don't know if there are lots lots of tokens around or not um again that's back to pan drops to to what they've got there but um you know you can't it's not a free for all like is on aleski you can't you can't spin up a validator without permission now we can obviously the existing validators can test consolidation or but yeah okay that's changed like 1,700 validators in total or something so you know it's it's a very different Beast to heski in terms of who contest and how big they contest it will still not be possible on seia because you need a total active balance of 18 million e which is not fulfilled on yeah okay okay so we have nowhere to test consolidations

currently not at scale and not general purpose yeah yeah I think some was tried on Emory and seemed to be mostly okay but um the tooling was a little limited from what I hear so I'm not sure how confident we should be if that's the only test per it unfortunately the same story there we have a twoo small validator set on Emory so consolidations are not active there should should we be considering a devet 7 with one and a half million validators on it so we can go back to solas's point of testing at scale and also being able to test large numbers of validated consolidations and similar we only need like 800,000 Val or

something we wouldn't need one and a half million is it practical to make a 800,000 validator debet like is there a reason why we can't we've done such big test STS before so yeah if you decide to do so I see no reason that we can't I think it would actually be quite valuable like this seems to me easier than the whole like revive heski from some old snapshot and Shadow fork and have like a bunch of the validators out of sync um so I yeah I don't see a reason why we would not move forward to do like I say call

it like a million validator which is roughly the same as net rate and it's like like 1.03 or something um so yeah I think if Panda Ops has some spare Cycles selling up devet 7 with a million validators um to do like a replacement testing from holeski would be good and that seems like the minimum bar I don't know if there's other aspects of heski that we'd want to test on a devet that we're not going to get from sepolia obviously the devet state will be kind of tiny compared to a test net but sepolia has like a larger State um is there anything else except from the

number of validators that we've like quote lost I think the missing State could be a big problem because there are quite many staing pools and projects that would like to test on Petra first and we don't well well of on theoa there's a closed valid set so we can't have these staking pools there but if you launch a a separate death net we won't have these projects there yet so can we they they could they could join right I mean they can deploy the smart contract they can run stuff I mean the the other trouble though and this is one of those typical testing issues we could reduce some of the parameters on

devet 7 obviously you can't consolidate until after 27 hours or whatever it is etc etc so there's a bunch of of conditions around consolidations and exits and similar we could tweak those figures but then obviously we're not testing similar to main net um so if we're doing that we would end up in a situation where devet 7 would have to be relatively speaking long lived we'd have to be looking at a month or so because people would want to deposit validators create the validators run them consolidate them everything else um but obviously that's not just one entity that's a lot and we're again you know limited on the entrance queue um so we probably need to recognize if we're spinning up a devet 7 and doing this in place of heski then you know it it's

going to be running for a while before everyone's done the basic testing that they probably feel comfortable with doing is there a way that we can have Genesis validators that are like quote transferable like so I understand like I think this is a good point where um we we'd want to have like say staking pools or DVT operators or whatnot running um and we don't want to necessarily be bound by the entry queue so is there a way that we can just like reserve a bunch of Genesis validators probably the key right and we can we can give to people however yeah the issu is around things like DVT and similar where they have processes

through Lio potentially where they're feeding through smart contracts and similar so we we can absolutely Short Circuit giving people Genesis validators but if they've got certain processes they want to be able to test yeah then you know like we said if if we if you know let's say a Lio equivalent or something where they have a smart contract they want to feed this through and off they go then we can't test that but I mean we we we're probably still going to be 80% of the way there if we just have a large devet 7 and then hopefully the entrance queue is much lower so you know the validators that need to activate through processes smart contracts whatever will not necessarily get flooded by you know some random guy wanted to run $1,000 valid dat to test yeah so I think okay what I would

suggest then is we do devet 7 would call it like a million validators have devop set up a large set set of those validat say like you know one 200ks that are somewhat easily transferable to like known entities um also give a big chunk of e to a bunch of addresses and them to like see their validators um and this won't be perfect but it'll allow sorry it will allow us to test a large validator set from day one they have a way to transfer already activated validators to like uh you know infrastructure providers and sticking pools and whatnot and then have a bunch

of East that we can send to people through like some faet or just in larger chunks and to actually do like more complicated set setups um does that seem reasonable okay and yeah sorry for D sounds like a plan yeah yeah sorry Maris you were saying something I said yes um okay we only have three minutes left anything else people feel we should cover before we wrap up yeah parents it would

be yeah sorry I have a quick question are anyone aware of any relay and Builder that's currently working on holes right now meaning are any of the blocks has proposed today are going through exal relay and Builder so sorry uh I think flash Bots may be running a heski relay and is went from kill on a call uh Sebastian anyway I was talking to Sebastian I think the bad block that had the deposit that started all of this came from uh the flash spot relay okay no worries yeah I don't know

are you looking to test it or something no no no no uh local block is fine I mean we just need blocks so local block is fine but then again I'm thinking about like if the not this up isic I don't need to wait for the E to sink to the head I just use the Builder block and that's even faster but then again there's definitely some complexity here it's not worth getting into it right now but yeah we can talk about this offline yeah are do bers propose when they're optimistic I mean we're not supposed to but we can hack it to propose Faster by then again I see I see a lot of debate here yeah yeah I would we can cover this many of these types of things we do

strictly n terance okay um and yeah last Maris had a comment saying you know teams should write up their learnings and postmortem and all that I agree that's all very important but I think we should keep like let's find to wait a week to do this property let's get heski back on track um ideally for cipia um and then we can uh um yeah we can reflect um and yeah Paul to your question I

think yeah we should ask this question tomorrow I'm not sure we'll have an answer tomorrow um yeah should at least open the conversation and if we can have a clear plan to move forward with sepolia and a new Dev nut and that's already good start anything else before we wrap up okay well yeah thanks again everyone for the hard work getting us to the space uh yeah I think we have a chance to save heski so let's try and do that and keep the chain alive than thank you everyone Che guys

you take care thanks everyone