Dencun Interop Testing #39

Transcript

okay we should be live um welcome to what is our 39th uh testing call since the start of 4844 um uh yes so the I guess the main thing uh I had on the agenda was just if there's anything on uh sepolia we want to discuss uh but we feels like we've covered that on ACTC last week um blob expiry have we seen that already on Gordy and then anything to figure out for Holi and if there's other things people want to talk about we can obviously cover those um but I guess yeah starting with blob expiry does someone know um when we will

see the blobs expire on gordi I think it was Wednesday but let me just confirm when Wednesday in two days right yeah uh but I could be wrong with the date FOC was on the 17th right I I am not sure if it was the 17th and 18 days was yesterday so we should be seeing the Oh you mean oh sorry the fork was the 17th

yeah the girly Fork yeah um yeah let me just double check it was the 17th yeah yeah so it should have been expiring as of yesterday is there a way we can check yeah we can just query one of the older blobs right and then we should be good to go I can just do that now awesome um but wouldn't it be sinking where we would really want to make sure things are working correctly like if I'm bootstrapping a new node now I want to make sure that I'm downloading the blobs that haven't expired

yet yeah we'd have to start sinking here right but we can have so we can have that done before um AC this week yeah that would be great I guess just syn one of every e and I guess the e doesn't care is anyone planning to run a non pruning C for mainnet is that exist for blobs

does any client team have a non pruning option yes I was in mute okay yeah I I remember one and I remember we implemented um a flag for non pruning for not pruning blobs but I need to get it back I don't recall exactly yeah remember Lighthouse had one too lar uh is implementing a on pruning Flag Great prism can have a longer printing window but we're not testing that right now either okay yeah I mean it's not a high

likelihood but it would be good if we knew the blobs are being saved by a trustworthy party yeah because eventually I think we need to have a better picture about how are we going to save this for the future and share with people who are interested in getting up to date with historical buls do we know what uh does blob scan keep the blobs forever or did they just keep them for the rolling period last I hope they keep them forever but

we should definitely confirm that uh we're also storing all the blops on our sides out who just tosses it in a database so worst case we can pull them out okay okay sounds like we have the bases covered yeah um sweet yeah so yeah if we can have uh some sync nodes by Thursday that'll be a valuable data point um anything else on the expiry uh Justin pinged me with what

seems to be a memory issue on prism that cropped up around the time we started expiring so we're looking into that today okay nice uh we had the blob s The Blob spam test from the nethermind team on Friday that caused a lot of chaos yes new yeah uh Marson do you have updates on that uh yes I was like spamming with a lot of one blob transactions like uh about 25 100,000 transactions uh uh and uh some some clients didn't uh handle it well but

they like found the uh the reasons and and seems like they are fixed because today I'm spamming again and as far we have no issues so I I will try to break it break it again but for now we are working fine oh that's really good news yeah so for for us we had a we had a o um basically whenever we created a block we pulled up a bunch of transactions from the from the transaction pool and also pulled up the blobs and instead of dereferencing them we kept them in memory somehow and then

so whenever our node created a Block H created a block we would pull up a bunch of flop transactions and just like allocate like two and a half half gigabyt of memory within a very short time and um yeah then at some point just ohm and then uh then yeah the note shut down but that's fixed now but the root cause was just that earlier all the spamming was with six blobs all the time and now we did it

with sometimes one and sometimes six or what actually changed because we've been blob spamming this whole time just lot of transactions I think us a lot more transactions than before just a sheer number of transactions not the not the block the number of BL fls yeah so what change like uh there are different implementation in clients so in nethermind we have a cup of amount of transactions for we are capping it to

16,000 uh so like the worst case for nethermind is 16,000 of six block transactions and I was always uh doing it this way like to to to check uh uh like uh how nethermind is behaving in worst case scenario and uh like few days ago I started spending uh more one block transaction because like if there was only six plop transactions it was like in NE mind we have 16k and in G it was about 13 14,000 and it was just they are copying it by size like

to 10 gab from from what I know so it was lower amount than in neite and when I started sending one blob transactions then in nethermind it was still 16,000 but in G it was like 50,000 and more so amount was way higher and it is why it is why we catched it uh at this point and not earlier okay oh hi

Barnabas anything else on the spamming yes I'm continuing the spamming like for now devet looks fine all our clients are working fine I'm planning to test some replacement and send some invalid transactions so I will keep trying to to break it but uh uh like for now it's all fine I just tested on one of the nodes for expiry and it seems to work fine as well um of course we still do the syn tests but at least it looks like Tech has expired the blobs as

expected um for this Epoch I just run this request and to make sure that the request itself is valid um here's a newer Epoch and a newer request so the older one doesn't give me any data and then you are one data nice um and yeah B we are spamming

again uh how many blobs per slot what what is the target because I see most of the slots are empty uh yes but it's like totally different scale than than earlier and today as far I sent 2000,000 one block transactions and uh uh now I will try to to to send replacement like the same transaction that we with uh higher prices so the the amount will not be higher today but I will send like

uh transactions for for the the same address and nonsense again with higher prices yeah currently it looks all fine uh looks a lot better than last week at least also just want to mention if someone has validators on hold key please update them yes

um yeah do we have a sense for how many validators are from non client teams and like people we have no idea on hosy I guess permission validators yeah it's it's a small minority it's only a few thousand got it I think we know the numbers to like 1.46 million and then there's another 80 90,000 that came on later okay but most of them requested e directly so already reached out to them to ask the update okay got it yeah and we are also going to do uh some deposits and exits today and tomorrow

and uh hopefully it's going to be like 5,000 and then hopefully we're going to see the turn limit being activated on hosy in real time when the fork happens oh yeah maybe we talking on that a bit more initially we wanted to see the Chann limit on gly the issue is that someone start exiting in Mass so that negated a lot of the deposits we made and the more exits that are deposited the longer it would take for us to hit that number where we churn um I think we're still about 2,000 validators away but there's about 5,500 deposits and 9,800 exits already so it we're going to continue trying but at

this point it looks like we'll get an answer way faster from hsky than we ever would from gley got it anything else I guess on so we sort of covered a bit of of each of uhia the blobs and host key but is there anything else on the test networks people want to bring up okay more confirmations as well that uh the uh G blobs are

expired uh maybe we can discuss whether we want to do a main Shad fork with the final may not release candidates and the timeline for that we definitely should yeah I guess we expect to discuss a timeline for minute Fork on Thursday yeah I think I think assuming yeah assuming all the issues from the spammer uh are are are fixed and like y clients are not having any uh issues um yeah we should be good to go and then yeah the other thing like Perry mentioned earlier having synced a

couple new nodes on Gordy to make sure that syn still works post expiry I think those are the two main things and uh yeah then um yeah on Thursday we can agree to a date as soon as we have the releases come out we should uh yeah we should run a shadow Fork which one small question about the syns um do you do the CL teams want me to just try uh checkpoint sync or should I also try Genesis sync because Genesis sync will take substantially longer Genesis sync will require you to enable some flags of like enable uh non checkpoint thinking or whatever for yes for and Lighthouse also

load star also so yeah can we do checkpoint syn with a checkpoint with expired blobs like will that be that by we use like the latest yeah we'd use the latest checkpoint yeah which would be the default we recommend to people anyway yeah but yeah I guess Genesis now would fail yeah Genesis sync would technically fail because unless someone has an archive Noe on good C arive there probably are some I would assume so at least we could give it a try actually yeah might as

well so then if we agree on a date on Thursday then I expect client releases to be around on 15 16th of February and then maybe we can plan a shadow Fork yeah on the week of the 19th yeah I think that that probably makes sense let's obviously confirm that on uh on Thursday's call but I think that's reasonable yeah okay yeah because we do need some preparation for that and thinking some minute nodes and that will take some time especially for archive

nodes okay yeah then let's yeah let's from the devops and be ready for the 20th like it's easier it's it's better to be ready for the 20th on the devop side and then push push it back because whatever we're missing a client release then it is to have everything ready and then be waiting for notes to sync yeah yeah okay yeah and then um the 20th also means we would have like a testing call

the day right before so we can uh discuss anything sort of final last minute issues right before the shadow Fork so yeah that seems reasonable um so related to syncing I think someone briefly mentioned whether we've tried checkpoint syncing when Bobs are expired um so how it would work is like since blobs don't expire for 4,096 Epoch and we always checkpoint syn from a finalized checkpoint that mean you can't the chain would have had to not finalize for 496 EPO and um that's like around what the weak

subjectivity period is which is like the period where it's safe to checkpoint syn so when we were talking about this like a few months ago we're talking about how clients would in this scenario have to support checkpoint sync from a non-f finalized checkpoint and it would have to be within the blob expiration period um and so it's actually like a pretty tricky thing to do but it's it's an unlikely scenario um so in Lighthouse we're like we're still working on getting that working just just so everyone knows I think it's not just Lighthouse uh last week we experienced the nonf finality event on devet 12 and I tried to sync a load star node and I also had

to sync off of another no not off of checkpoints thing I think that was related to this too no that was related to checkpoint c not serving it and un finalized it it's not to do with expiry yeah cuz this scenario would only happen if the chain hasn't finalized for like three weeks or a bit more I think yeah oh okay okay um anything else people feel we should

cover okay questions if not um yeah let's wrap up um and yeah let's uh discuss uh main updates hopefully on ACDC this week and assuming that moves forward aim for a shadow Fork uh just over two weeks from now yeah thanks everyone all right see you on Thursday thanks guys bye yeah all right