Pureth in Reth: SSZ Execution Blocks and Query Language

Transcript

Thank you guys. Part Arsh, thank you for being here. I hope you guys are ready. Uh, thanks for making it all the way here and I'm curious to hear about your project. Yeah, go for it. Yeah, thank you Mario. Uh, so our project is also pure in death. uh and uh we we worked on two projects and uh we are going to present together as we said it's a DP so we worked on two projects uh uh the first one is SSG execution blocks as uh Vive presented uh it's the same project but uh but we uh did it in red and the second one being SS query language uh which is uh the uh the the proofing and uh query pipeline for that

so We have divided the PP into uh sections and we'll be taking turns explaining uh ours. So uh please bear with us. Let's get started. Uh yeah uh special thanks to our mentors Eton here and Tamagna. Uh Tamagna was not able to join. So a special mention to him also. So let's get started. I'll just briefly explain what Pureth is and what were our two projects. uh pure uh the EIP7919 is uh is a wider effort to like uh to make the accessibility and verifiability of execution sources uh easy and more efficient and uh in it as I told uh we worked on two projects SG

execution blocks and ended uh in it we worked on uh like three different execution objects first one being withdrawals uh which is the EIP 6465 5. Uh second one being the receipts 6466 and transactions 6404 and the last one which is the main goal of the project which is 7807 SSG execution blocks and the second one being SSC query language which is again the uh querying and proving pipeline for that. Uh so I'll be talking about how we approached the project in breth because uh he Vive has rightly explained the technical stuff. So we started with a

prototype. Uh so first of all why did we start with a prototype? Uh as you all know pure is like a really new concept in the execution layer. So we started with a prototype and a schema which is uh not the exact uh EIP 6466 and we omitted some sub fields uh in the starting because it would uh make like establishing the whole query layer uh e easy to code and uh easy to test as well. So that's why we started with a prototype and uh this was uh the first path which uh which we tested on and uh yeah uh this is how our scope changed throughout the project. Uh we took a

bottom to top approach like we didn't start with the client integration. Uh first of all we uh started with the execution objects uh the withdrawals, receipts, transactions and after that uh the blocks and after that we moved on to integrating all that into the rest code base and uh yeah we had our prototype at the start as I told you. Now coming on uh to the three objects I was talking about uh the first one is withdrawals. So why did we start with withdrawals? because it is uh just a simple strct with four sub fields and uh it was like really easy uh to work with and uh we just had to convert it into

the uh progressive container and like pour some test around it and it was done and uh the receipts was the part where uh actually most of our time went because we worked on the prototype first of all for that and then we switched on to the exact receipts uh which has uh like three sub fields uh basic create and set code receipts And u in that uh in the e official EIP uh descriptions uh there were some uh cases like edge cases which were missing and uh we had uh like uh our communication with Ethan. He had a call with him multiple times about that and I talked to him and uh uh he told me to go and study like how is it handled in red

and uh I studied about that and u come up with a came up with a solution and uh we uh discussed about it and and he told me to like uh include all that in in the official description of the IP. So I opened I opened a PR for that and it's currently under review. uh we are like uh going back and forth with discussions with Ethan on that and uh after that we moved on to transactions. Uh there were like in the main transaction payload there were 10 sub fields and uh it was really easy because uh after receipts we had uh like everything set up. So we just had to convert uh the existing transactions from RLP to SS.

And uh this is like a simple diagram of uh uh whatever we introduce newly to the code and uh how does it uh uh like sits together with the existing codebase. So we introduce uh three directories. First one being rest pure node and uh it uh it combines our project to the existing uh rest code base and the second one is rest block which is like uh our implementation where our implementation implementation sits for the 7807 and the third one being uh restp query which ar will be talking about and uh uh it is like as I told the uh query engine for

all all we worked on. So now to explain uh more on the payload summary and query part, I would like to hand the mic to us. I guess I only have a mic. Okay. Uh hello everyone. So yeah, it's a double presentation. I mean uh 70% cut I had to show somewhere. So um let's start with payload and query. Okay. So for payloads, uh it's basically carrying all those transactions, receipts and withdrawals. Uh then uh val bytes also of course boxes list byes and u the summary summary basically replaces those components with roots uh at their matching respective tree positions and

um the request and roots are receipts are already shared because uh those two be have to be encoded uh and have to be matching tree roots in order to work and uh that is also uh in it is the case because the compact summary still commits to the payloads component. data uh and that is because uh so the caller will be able to check uh that whatever it is calling uh is verifying with the expected route. Uh so that was it about uh and the commitments. Yeah.

uh it has a lot of uh commitments basically you can say gas commitments to gas fees base uh used uh or the limits uh you can check that with the same route um type I can also see it here right uh type commitments for EIP 7807 byte entries basically it had it helped us to have the byte entries uh uh in the encoded formats so that what is that was it about the IP role and um encoding and validation. So basically uh while decoding we had to check the offsets uh fields and roots uh and that was it because uh uh to to c to catch some of the errors

like uh encoding creates some errors. So uh if any commitments mismatch or uh any image m any sort of problem happens so we'll be able to catch that. Um uh and then the retain recepts and body requests too. Yeah, we had to do that because the roots do not alone provide all the data, right? Uh so yeah, that was it about payload and summary. Uh okay. So let's move on to the main queries and proofs uh pipeline.

one source for uh uh one source for values and proofs, right? Okay. So, we actually kept uh these block values and proof C together because we have to fetch the data, right? Uh so, it had to be consistent like we cannot query other and proof uh fetch other data. So it will be like a total mismatch about the data and then uh the same source helps because later query will also be able to use the same thing instead of construct reconstructing the hashes or reconstructing the trees. So it will be an easier job for it. Uh and then the verify yeah the basic verify part that uh the caller will then use the selected proof uh to check the answers of the

root it already trusts. So yeah this was it about the slide. Now at Quenry query uh we actually started with a basic simple prototype that uh even part told about uh but then we expanded it into broader selections like for the basic things also uh we had bytes and range slices uh we had the presence checks or maybe you can say we had the whole objects or the supported values itself and then we expanded it into the execution objects that we had like the withdrawals, transactions, receipts and logs. So basically we came a long way from uh the basic address uh recept syntax.

So yeah this was it about what we can query. Uh now several selections one proof. This slide basically tells you that uh yeah we also have multiproofs. Uh and yeah you already guys know what the definition of multiproofs is. So it is basically a representation of showing basically showing you that how exactly are the query syntax here that we have. uh like you can group these in uh together in a multiproof. I believe the syntax was something of the selections array and that had these fields operation and uh the kind that we wanted false uh uh false and the values returned would be the uh expected answers.

So yeah, this was it about uh selections. Okay, now we have uh connecting objects to the block commitment. Okay. So we have to like prove that a value comes from a particular set of answers, right? So uh like we have to prove that this data belongs to this particular set of uh things. So for that we have the field proof and then the link proof. Basically link proof helps you to uh commit all the data to the summary route. So uh almost all the things can come from the summary route itself. Uh almost everything near to what we want to verify. Uh and then what is this code?

Uh it's basically it's basically uh checking for the link proof. It is that uh it should make sure that uh everything falls under the uh summary commitments that we have proofs on ordinary recepts. Okay, we actually had pro pure uh two extra additional RPC methods. Uh but uh we also had these uh SSC proof uh as an optional uh to the ordinary query. So you can just add this field and then attach the proof with a normal query and then when someone asks for this you can like it will be for one receipts or all the receipts it will share the same

source. After all it has to come come from the same commitment. uh and yeah then the caller checks those roots with the official trusted root uh that belongs to the block receipts or on that level uh transaction and receipt association. This is basically for like we prove that the recept is for the requested transaction like it has to belong together uh because we check those both at the same position in their lists. I mean uh they have to belong to the same expected block after all. Uh and then for the transaction hashes too uh we have to actually check those transaction hashes that u like we we check a lot of things because this is a proof pipeline

like we cannot trust the center we cannot trust what we are calculating. So we have to make sure we verify it from everyone. So even the transaction h is recalculated uh and it's made sure that uh it is what you are trying to expect it to be and the bound combined uh bound the combined operation. This basically means that we had uh all the operation bound together and cancellation checks. These cancellation checks are not for like we trying to save gas. It can be for trying to save gas. uh but for what I worked on it was mainly for uh helping the node to not be overloaded or basically able to cancel it in between before like if any error occurs or something fails so it

will not continue the rest of the work and it will save those resources and computation on the same route uh on the same node I mean so uh we can actually expand this to a lot of cases to make sure we optimize this code uh and then we're going to come to client and integration. Uh I'll invite part to the stage again. So okay. So uh now coming on to the last phase of our project which was uh like integrating whatever uh producer and query engine that we have introduced to the existing red codebase and uh this is where like we had uh majority of our problems uh in the implementation in the

project because uh pure is like uh a really new concept and uh uh there was like a lack of uh reference implementations at the start and uh to integrate all that and uh breaking breaking something uh was like a really high probability. So uh we'll talk about that. First of all, how did we decide to uh build and validate in red? Uh first of all like we had uh we had different for uh four crews like as uh Vive said we had to uh switch from Glaus to Electra because of uh some error that Eton also talked about and uh uh the second point

uh which is uh validating valid validation by the receiving node. So we uh built an execution source uh which uh which uh like consumed all of our uh producer implementations and uh it uh exposed that to the uh query engine uh so that uh the proof the proofs and the query it can uh use the data from the same source and uh after that like we have uh it was a design choice that uh we have uh built everything as an optional uh as an optional implementation beside the RLP RLP implementation which already exists in red like we haven't replaced anything and we haven't uh like uh introduce any

breaking code as of now. Uh so now coming on to how we have modified the block identity inside. Uh as I told you uh this is like a uh optional implementation. So after the activation of pure it can be uh seamlessly uh seamlessly migrated to the SSD pipeline and the existing RLP pipeline uh doesn't have to be uh like changed a lot. So and uh uh the second point which is exposing the same identity uh like we haven't really uh we haven't really uh changed the lookup mechanisms of the

block hash. we have modified how the block hash is uh like defined itself. We haven't really uh touched how it is looked up. So that's why uh the block hash and the existing EIB2935 implementations are the same and they are intact. So uh the last point is just uh observed uh result that we observed that uh before the reorg the hashes of the match engine uh block RPC and block hash uh they were as expected and what happened after the after the reorg we will cover that in uh in a later section.

So now uh this is like uh a highlevel imple uh diagram explaining everything uh that we have worked on the project so far uh and whatever we could fit on this page like we had two ref LH pairs running and uh they were like for the ELC communication there wasn't there was the engine API and uh to communicate between the two LH pairs uh they were exchanging at test attestations and uh for the query and receipt RPC it like uh consuming the matching values and tree witnesses and uh uh for the caller which was uh querying about uh the proofs it was handing over the values and proof that is what as it uh as it is written

here. So now to explain uh more about like uh how did we run the LH pairs and how did we run a devet I would like to again invite us. Okay. Now uh we had uh to run uh in order to run the process divet we had to make sure that we uh sort of run it with another uh u client then we chose lighthouse and uh uh we had to reuse some of the core items uh I mean we made our SS object and commitments uh reusable in lighthouse so like both of the clients would be able to use same calculations and uh like without needing

without the need of red's codebase uh that's database, right? Uh and then Lighthouse basically we made it to rebuild the uh execution summary and then check over those commitments and the whatever uh whatever we on on the other side were producing all the commitments and block summary maybe uh or the payloads. So yeah that was the uh main pipeline that we had to connect with lighthouse. Uh and then yeah we had these checks uh to for block production or the gossips or the retained history. We had a lot of checks but uh yeah these uh can mainly summarize uh it uh and then yeah of course for the dev net we had uh built new blocks and then we check the incoming network for the

traffic uh historical payloads uh commitment summary everything. Uh now for the devet yeah I was talking about this itself. So uh we had well next slide uh okay we ran the goods devet as I told uh and we had a modified breath and lighthouse together running as part told already as you can see in the screenshot that we had uh almost 257 epochs finalized and I believe we had more like 350 plus until we turned it off because it uh worked just fine. uh we used two uh rest LH payers to make to make sure that uh we had a Pier 2 in order to uh check send receive or send blocks or

construct them. Uh and we had 32 eleators which was which were having a 16 split. So uh that was able to help us check that blocks produced by one pair and then receiving by the other pair pair and how it was processed and all and yeah they imported blocks they produced the blocks and uh uh the finality just kept advancing. So it was good to see finally that something was working out so uh um I believe it's not uh often that we see something to work out at least at the end. And what we validated okay we validated a lot of things because there are a lot of things we can do here. Uh uh starting with execution commitments and proofs reox and restart blob data we had for

checking the incompatible proofs we had for uh like repeated logs transactions uh that the transactions that could change after a reorg. We had a lot of checks. Uh now quickly summarizing the benchmarking and your implementation as well. On the left you can see optimized build. Basically we tried to produce the blocks and uh start build the proof make the proof and then distribute the proof. Uh you can see the median timings at the 95 95th uh order timing too for that for reading the data we had these values without constructing them. Uh and uh we also had uh tried benchmarking the RPC response uh and it everything seemed pretty decent and we have to work more

on this I believe to optimize it. uh now beyond the cohort I I assume that we we'll try it with more clients to see how that uh responds because we are not quite sure how because as part told that this is uh actually new sector and uh yeah we'll see the responses with deeper yog data and availability and we'll we'll add some more implementations to the code itself uh and the backend migration uh the has also changed its execution engine from revm to EVM2 which is not a big thing for us to change but it will still require for us and uh yeah all those uh recheck commitments and proofs. Now the SS engine representation and transport we were we only were

having the JSON engine support for communicating between those clients uh for the ktosis devet and and we'll probably look it on to more SSD engine but I believe we had this added because we had to fill the PB2 I mean yeah okay so now coming to the acknowledgements uh really thanks to Mario for being there at all times like from smallest doubts to deep discussions and being there to help us every time. And also Josh, I I'm so sorry that he's not there with us, but yeah, he was there in the start and he was so cheerful with us. Uh and yeah, especially Eton. Uh I mean, man, it's not easy to review like 3,000 lines per

week and then comment on all of those and then re-review those feedback PRs from me and him because we had combined the project and it was like double of the code it should be. And then the Magna who is not here with us but yeah he is he has guided us a lot in the starting as well. Uh Matias is also not here which is who is the maintainer but yeah he has guided us uh with the setup and breath. And to my dear fellows uh I believe uh we are here from all parts of the world and uh yeah we had pretty good conversations and I hope this stays with us more longer uh and all the team that is here helping us out and many many more. So thank you everyone. I guess that is it.

>> Thank you. Thank you guys. Um, thank you for our first double presentation today. I enjoyed it. Thank you guys. Um, do we have any questions? Any thoughts? Anything you want to ask before? Otherwise, we have the 2D filter maps in red presentation now. So, feel free to join us. Ah, we have something. Yeah. Uh just a quick one. Um you mentioned that this SSC stuff like the bite count goes up, right? Um like it's bigger size of the objects. Um did you verify that before or after it gets compressed like because it also goes through snappy

right in a database? Uh uh can you repeat what in exactly we have to >> like when the transactions get stored to disk they get compressed with snappy at least in G I think um it's probably the same in ref. So the size comparison did you do that in before the compression or after? >> I don't think it gets >> I don't think we have uh like worked on this for now. We'll definitely look into that later. >> Okay. Okay. Okay. So I guess >> we did not hit that compression part for the transaction and >> okay okay so so in the database it's still stored in RLP.

>> No no >> also the one >> it's also SSC. Okay. >> Yeah we have uh we have modified the database for it to shift to SS. Nice. Nice. >> Like yeah, we have all the fields explicit and like all the schema and uh data types explicit. While in LLP it was just by strings and lens like even the query had a lot of