Stateless Implementers Call #44

Transcript

Loading developers...

All right, let's get this started. So, welcome everyone to stateless implementers call. This is number 44. Uh, so the agenda for today um is quite small. We've had a last minute adjustment because uh Carlos is sick and unable to take his point number two. Uh so we just have two points on the agenda today. uh starting with team updates followed by compression expiry which is something that GM uh wanted to talk about. So let's get going with team updates. Maybe G you'd like to go first.

Team Updates: Discussing test net launch and debugging issues

>> Oh, I was I was hoping to be last. Um yeah, I spent some time um discussing with Karim like that was mostly today. Uh we're trying to to get the the test net launched and make sure we we do have interrupt. So I'm sure Karim has a much better um uh yes series of information on this. Uh what Carlos did Carlos not being here. Uh what Carlos did was to generate the tests uh with the branch like what what we discussed last time we have this branch with some tests. Most some of them actually are not completing but some of them are are working and because of what we discussed last time

we want to uh rebase or uh port our changes to the new version of EST with all those uh with this new repo and and everything. Um we just like we just thought let's just generate what we have to see if we do have inter problems uh and and see if Karim can consume this. Uh well car once again Karim will give a a deeper uh uh update on this I I'm sure. But uh the general idea is we do not have interrupt. Uh, and that's kind of weird because uh we had some test vectors generated before that uh that uh seem to still be correct as far as I can tell, but uh the the

test vector that uh Kim extracted are not the same as what guess is getting. So uh we had a bit of debug. Looks like it's an uh RLP encoding thing, but yeah, uh I just shared the the data. So we'll uh the investigation is still ongoing. Um I think in terms of um of an update uh yeah we've done a lot of state expiry but maybe Han wants to give more more info about this >> and maybe you want to take it forward or Karim.

>> Yeah I can. Uh so for the month still investigated so I don't really have more information. Uh but yeah uh I received the d the dump from gu. So I need to look why it's different. Um yeah don't really have more information. I saw that game said it's European coding issue. So I still have to look on the transition side. Uh we had a meeting with Thomas. So we are trying to see how Toma uh yeah can help on this subject also uh because it's uh it's moving forward but not really fast. So yeah we will see if can help also on this subject and

what we can do also when we have the transition maybe to measure the impact regarding the size database the performance etc. the it will be really nice uh when the transition will have will work to have all of these metrics. So it's also uh something we want to do after the transition feature implementation. So yeah this is mainly the update on our side. So for the next step yes trying to have the the inter between get and vis for the binary tree.

>> Perfect. So the discrepancy that you guys have uh encountered uh is between get and beu. >> Yes. Yeah. >> Good. And then have you had the chance to test with other clients or is it those two that are impacted? >> I don't know if you have other clients with binary tree. >> You maybe you know I don't know. >> Yeah. Yeah. Um I I don't know. Last time I talked to Tanish, that was a few weeks ago. I'm trying to remember exactly what he said, but um basically maybe maybe Lukash knows. Uh yeah, I think it's been mostly on the back burner.

Um and uh yeah, he wants to do it eventually, but I think he also wants us to to make some progress to to see if it's worth the time. >> Good. Thanks uh Kareem. Anyone else on the for team updates? I can give a quick one onJS. Uh not much but we have well unfortunately decided to deprecate vertical support since it's

it doesn't really have traction. We could always you know recover it back if uh if needed. Um yeah I had been the one working on it and I was the one to also remove it. So it's a bit always a bit of a the sad thing but uh we're still keeping in definitely binary trees although our initial plan to support as a stateless client is not uh you know phas feasible in the medium term. So it's still up for debates like in in what fashion we're going to be able to help probably not as a an actual client but could help you know run blocks and and some test vectors. So that's it for us.

Were there Yeah. H >> sorry. Um I just wanted to give my updates but um >> maybe I let you finish first. Yeah. >> Oh, I I was finished. I was going to open it for others. >> All right. Thanks. Yeah. I'll I'm just going to give a quick update on side of CSR. So there's this uh article that I just posted today on um basically comparing between in protocol versus out protocol because um I mean I mean the whole point of doing this is that um some people think that um state expiry is something that should be done now but in if you look at the long term with real time proving with um like stateless testers u maybe state is not really that important in in in some people's view.

State Expiry Discussion: Article and ideas on contract-level expiry

So, so this article basically outlines the role of stay as expiry in today's world and then we look into long-term and some of the challenges that that we face and some of potential solutions that we have. So, um do do give it a read if you have the time and let me know your thoughts as well if you have anything that you want to share or discuss. Um so other than that um I think internally we have also been discussing a few items on stay expiry. So one of the items that um we we are talking about is I mean at least from my side is something called like uh we have this idea called contract level state expiry. Basically we want to expire uh we might want to expire contracts storage only in a sense where if a

contract doesn't have storage activity for the past one year then we can remove the whole storage. So that means we uh e clients can also remove the storage try. Um so that's basically the gist of it and we run through the data and we found out that we can expire about 20% of the state size. So if you take a gaff node um that would be removing around 50 to 60 gab bytes of um storage space. So that's roughly 20%. Um yeah so that's one of the ideas. I mean uh I would lay out all the um specification in in EIP and maybe also bring up again to discuss uh in more detail but uh that's one

approach that we have but um other approaches are like for example compression expiry that GM will talk about later. So yeah, >> good. I'll piggy back another update on on that topic. That's something uh I'm focusing on. Uh had a discussion with Guiam and we'll create u a survey that uh that we will forward to individual you know wallet or app teams to try to get a discussion going on state expiry.

Uh some of the things we want to ask is are would projects be open to you know be responsible for some part of the state that concerns them or their app? Uh what would that look like? What are the costs associated with that? What would be the potential incentives that they need to make that happen? uh we want to get the discussion going with these teams you know as soon as possible and not just you know push something that eventually gets you know push back or you know support by teams we want to bring them to the discussion they might have some great ideas to contribute as well so I'll be focusing on you know first getting in touch with all these teams and writing the survey in parallel and I'll share it with you guys on

telegram whenever that's uh ready for a view. Any other team updates before we move on to the next topic? >> Uh I don't I assume it's not a uh stateless specific update, but uh maybe Mario can tell us uh you know what's the progress of uh of that rebase. see how far uh okay so that we can guess how far away we are from uh from using the new the new structure

by re you mean um the process of welding uh >> exactly sorry welding yeah >> still in progress um we are looking forward to trying to complete most of it this week and it's in a good spot but um yeah I would say that at the very least this week and probably the next one should be like when we're uh finishing things up. So yeah, it's it's a work in progress basically. Nice.

Um so once this is done, this is when we can try to plug in the I think Carlos said he would do it. uh try to plug in the the binary tree inside the inside that that thing and uh and try to fill the test from from there. >> Yeah, that sounds about right. Um I I'll Carlos will probably need help from us to make sure that everything I mean because many things are changing in the repository, but it's not it shouldn't be that complicated. Anyway, we will try to uh give him support so he can finish this uh this part after we weld it. Yep.

>> Nice. Thanks, Mario. Yep. Uh all right. I think we can now move on to the next point on the agenda. This is number two, compression expiry. So yeah, Gom, I'm handing you the Mick. Oh yeah, Gom has a presentation. >> Guom always has a presentation, sir. Um yeah, so this is this is an idea we've been talking about. It's pretty early. Um the the whole point is to to collect early feedback. Um for your information, I am currently writing a an article about this. It's currently in a pretty

Team Updates and Project Progress

ugly state. So I don't want to make it public just now. But if uh you are interested in the topic and you want to to provide better feedback, I can always invite you to the to the note. Uh simply yes, I think uh I need a few more days to to get it to to an acceptable state. Um so we have this thing called compression based expiry and uh the name is perfectible but the idea is the same as the ancient store for uh for block. So this is what we we did in um in guest and I'm sure other clients did exactly the same thing. Uh we checked uh the what's that called? We

check the the output of the sorry we we take the old blocks from the database and we move it to to a flat file. Um and uh the idea is that this way the IO gets better like the size probably doesn't get that much better. Uh although we do remove the indices we can try some other compression algorithms that are not available in the database itself. Um but uh yeah it's uh like the idea is to is to at least get rid of one of the issues of of database of putting a lot of data in the database which is the the speed of of reading.

Um and uh yeah the the other advantage of this approach is that it's uh at least in its first naive implementation you don't have the problem of finding the data. you don't have the problem of resurrecting uh figuring out how to do the resurrection and the uh and the UIX of it especially um because every node still has all the information on a disk simply uh it's put uh into a colder storage where the data is uh is not um yeah is not getting in the way of faster performance.

Um so yeah as always there's a a weird diagram representing the state. So in the first version something that I haven't written but in the first version we would only expire accounts. Uh so that means that if an only one storage slot uh of a very large contracts gets updated the whole contract does not get um does not get uh um expired. Um okay sorry I just saw something from uh we we would al only put leaves data there accounts and storage slots. Sorry I don't quite understand the question.

>> So when we would expire something we would expire accounts and storage slots or would we expire like the three nodes? >> Uh yes. So presumably we would only put uh the we would actually put uh only the only the the um the values. Yes. So we would not store the the tree notes. Yes. And we would only expire um like the the account would have some kind of marker. I mean I'm going to explain but yes the idea is to is to get rid of the of the three nodes. Um the problem being that indeed uh if you uh if you write so I'll get back to this in a minute but uh but yeah the idea is to to remove the three

nodes unless we can't uh just do it. Um anyway, so yeah, like just explaining the full uh the full uh thing. Um you take some stuff out of the database and you uh you put it to the archive. Um and you put a little marker here to say there's something, but we we moved it to the archive. So some kind of pointer to say if you need it go to that file and that offset and you will find find the data you're looking for. Um the res the resurrection part it's quite nice because if you want to read the data

um you have nothing to do. uh if you do uh want to write you need to reinsert it in the main tree which has the potential problem uh that uh if you don't store the internal nodes uh you will need to recmp compute them that's a potential problem. So if we can't go around this or maybe there's a threshold where we say well if it's you know only 10 leaves uh we'll recmp compute the state route on the fly. uh if it's uh you know 10 million leaves we can't do this. So we'll stay we'll store some internal intermediate nodes so that we can we can do uh something like this like the design space is still a bit open but the idea is to indeed save uh like have a

good tradeoff between uh something that's fast enough but doesn't have to be too fast and something that is uh that is not too not to I mean that that serves some purpose of deleting building data Um right so like I said it's just about reinserting so that there was the tree was looking like this uh we want to re uh re uh like we want to write to that piece of the state we just reinserted somehow and it's available but the rest of the the rest of the uh things that were in the archive are still uh in the archive. Um and so at some point uh like this is

meant as a first step to a potential enshrinement where we just say well look this data is there we know it's there um but we're not going to uh care about it anymore and if you want to access it uh you're going to have to do a lot of efforts that are probably not very user friendly but you you haven't touched that in forever so uh we're not going to we're not going to go out of out of sorry out of our way to to get rid of to sorry to help you. Um, one thing that to understand is that this enrinement might not be voluntary if everybody deletes the same piece of state. Uh, well, it's just missing. So, in that case uh it's

the def facto in a protocol state expiring but not not intended but happens anyway. Um and you can also imagine of doing uh this process uh in v various uh epochs in which case you could say well look uh everything that has been that hasn't been touched for more than uh more than I don't know 10 years random number will be deleted and uh and uh yeah if you want it good luck with that uh you have 10 years to to react so So that's uh that's something um so yeah it all of that is a bit uh unspecific of course we're still working

State Expiry Strategies: Discussing methods for expiring data

on the details but the idea is that it offers a path towards something more in protocol while uh not getting all the the friction from having to do something in protocol at first. Um the question is uh yeah how do we figure out what to expire? Um there are two ways to do it. Either you do it dynamically. So you you go and mark every account uh that has been touched in the last uh or you write down when an account has been touched for the last time. The problem is it's going to create a lot of uh extra data because you need to store a time stamp. Uh so it might not save a lot of space after all.

Um but uh yes after that you can do a background process where you you do a sweep and you delete data uh from from the database you move it to to those files. Um and if it's just the accounts and not every storage slot I guess it could be done. So that's one of the things we want to measure. But uh yeah so far it's just it's just a hunch. Um and yeah otherwise the other method is the one that's actually on top of the slide. It's um you have to to go replay uh some historical blocks and somehow distribute it to to the nodes. Uh the

problem is that if you do this uh if you pop uh make those expiry list public then a potential attacker would know how uh what in all likelihood will be expired in the state and they could target those accounts that have been expired uh to revive them. So, and do that pretty cheaply. Uh, pretty much like the the prover killer approach where you just do an X code size on on every account and um and then this way you you resurrect everything when you don't really want to.

Um there's uh more problems with uh this approach. I mean it's the problem that is due to state expiry not so much this approach. Um, if we want to keep using Snapsync for a long time, uh, currently Snapsync will require all of the data. So, we will need to somehow, uh, tweak it to not download everything and Snapsync is a pretty uh fine-tuned thing. So, it's it's going to take some time to to get right. Um, and uh, yeah, if uh, you still need to be able to download the archive, which is not covered by uh, by snapsync. Um, so we could do it the arrogant way, but

that's very not centralized. Uh, so there's a question of who's going to serve it. And I don't know about the other clients, but the guest boot nodes are already hammered by all the requests they receive. So if they are the only ones serving that state uh well that's a critical point of failure first of all but uh on top of that it's more um um yeah it's it's a centralization uh issue to to begin with um so we can always expand things for v2 uh which means like try to expire individual leaves or u uh internal nodes um and um only partial views and uh some

um there's this article by Han that he shared earlier during that call where um we we notice actually I don't know if it's in this article I don't remember anymore but um yeah there's if you only if you only expire the at the account level you can save about 20% if you uh expire individual leaves you can go all the way to 80% % uh so the the benefits from doing this would be uh would be uh much more significant. uh nonetheless uh like I said earlier it's not just the it's not just the performance of uh sorry it's not just the storage u size

Transition to Protocol Changes: Considering in-protocol state expiry

that matters it's also the performance of the DB and uh and so just getting 20% at least in the in the benchmarks that I've seen on guess just deleting 20% is already going to help the performance quite a lot so um at least if we don't get the the storage benefits. We we get part of the performance uh benefits. Um yeah, I um I figured you would be saying that Wukash uh I do think given the amount of work um that uh that would be that would be done uh which is not that much uh 20% is worth the the work because there's

really not that much work but uh indeed this is to be seen as a first step towards the 80% that we that we want to uh to try to Um yeah so the idea or at least the benefits of this approach is that uh it provides a smooth transition towards potentially in the future enshrining uh state expiry in the protocol but at least it doesn't offer the UIX issues while uh offering exactly like I wrote moderate but immediate benefits um and uh yes sorry I just saw Wes's uh comments. I I'm underestimating the

the work. Um I don't know. I mean, okay, we just have to to implement to implement it uh to see uh to to get the the final word on it. But um I don't um I don't think I am un underestimating it, but yes, I I I admit I could be I could be wrong. And if that's the case, we'll we'll come back to it. Um anyway um yeah so uh what I wanted to say is forward compatible. So like I said we want to get 20% right off the bat uh and 80% later when we when we learn um and uh yes like the the end goal is that we still have to figure out

the UX issues which has been bogging down state for a while. So at least I would say that's that's a good step in in the right direction. Um but yeah, like I said, there's plenty of discussions to have. Uh there's already already one happening in the comments. Uh but um but yeah, uh so the thing is I want to publish it uh this week. Uh it needs to be reworked. If you really really want to uh not wait until uh you know, Wednesday, uh I can add you upon request to to the to the to the document. But it's also okay to to wait for two weeks for two days because uh

this document is going to be rewritten. Um I see there's been some more comments. Uh yeah, Han was saying it's always a tradeoff. Um but yeah, the UX gets better. I agree. Uh try is the hardest part of the code. That's that's true. Uh although I think think is the worst is even worse. But, uh, try is definitely harder. Um, but, uh, yeah. Um, I don't know. We've been we've been dealing with tries for a for a while, so we're just willing to to give it a shot and see, uh, see what happens.

Uh, okay. Karen wants to be added. That can that can happen. And yeah, that's pretty much it for for the presentation. So if you have questions uh I'm happy to take them otherwise uh we end it there. >> Yeah thanks GM and and yeah we can share with others at the doc once once you publish it. So the just as a small recap so this would be more like an out of protocol specification but would there be an in protocol change related to like how much uh when you're trying to write to very old like expired storage there would be an increase in gas cost associated with that right

>> uh yeah so if it becomes in protocol because if it's out of protocol there's nothing you can do but the idea is to slowly turn it and assuming it's worth it once again uh I'm not saying I have the all the answers on on this one. Uh but if it turns out to be indeed worth it in terms of performance, we could say well this is good enough that we that we want to enshrine enshrine it. So make move it in protocol and this way if you do want to recover it uh we could increase the gas cost which are going to uh uh to increase anyway.

Discussing State Expiry and Gas Costs

uh if uh you follow EIP 8037 or 8038, I forgot. Um and uh and so yes uh to be able to resurrect you could make it a um a uh I mean an in protocol gas cost and in fact that offers a way to do soft uh in protocol state expiry because you could simply say uh you could simply add some kind of mechanism to to write down what hasn't been used in a in a long time and if you see that you're writing to such location. Um, you could already increase the gas cost. So, you're not

formally enshrining state expiry, but you're you're signaling that you're touching state that is that is a bit old. So, >> yeah. >> Yeah, it opens a few things. Um, in the document I also wrote that if you're for example a builder and you do this kind of thing, you can also decide that you're not going to resurrect a transaction that uh you're not going to execute a transaction that resurrects something until it comes with a significant tip. So you could already signal still out of protocol, but you could already signal that the cost of touching something uh pretty old is is going to be more expensive.

Yeah. Yeah. Fair enough. Thanks. Any other questions or you know point comments related to what Gom has just presented with question state expiry. All right. So, it sounds like we're going to wrap this up early. Um. Yep. So unless anyone has objections, uh I'm just gonna call this off now.

Meeting Conclusion and Next Steps

So thanks everyone for joining. This was a stateless implement call number 44 and yeah, I'll be sharing a summary to the stateless book as well as uh we'll be publishing a Twitter thread sometime this week or the next. So yeah, thanks again everyone. >> Thanks. Take care. Thank you.