Code is Law: Avoiding Spec-ulation for Faster Forks - Revamping the EIP Review Process

Transcript

[music] So good morning everyone. Thanks Puja for the introduction. I will be joined afterwards by uh well I'm joined by Rahul who will um talk about code is law. So, we'd like to propose a small but significant change in the EIP review process, but would really help us couple the specs as executable specs and the specs as markdown specs from the Ethereum EIP's repository. And this would really help us accelerate Ethereum development and avoid speculation.

So, thanks a lot to Parry who um gave an excellent overview of the whole process. um of how an EIP hits mainet and right now we're going to focus right in on the stage of specs and testing. So just to motivate that a little what and why do we do specs for Ethereum and well the execution specs and the consensus specs repos they basically imp implement the intended behavior of Ethereum as executable code and these specs serve as a proving ground for EIP ideas. It gives a place to prototype them and they serve as a shared source of truth for that generates test vectors that can be used

by client teams. And for this talk today, it's very important to remember that we really need the specs in order to generate vectors. The specs are really essential. And furthermore, they serve as clear readable documentation. They're not optimized per for performance. The specs are really optimized to be clear and readable as a reference. And well, why do we do this? Ethereum's foundation has a foundation of client diversity, which is really important. It keeps the network secure.

And due to the fact that we have many client implementations, it makes sense to have this additional le level of infrastructure that provides specs and tests that can be shared across the ecosystem. So working in specs and testing is really an extremely collaborative job and we really connect teams across the core dev ecosystem from research, prototyping and production. All the client teams. Uh we basically help researchers and prototypers turn their EIP proposals into ri v ri v ri v ri v ri v ri v ri v ri v ri v ri v rigorous and unambiguous specs. And as I mentioned before, we generate test vectors and provide the

test frameworks that client teams can use to verify their implementations. And so we're in a position in the middle of many teams where we have a huge potential for impact. So we we collaborate with a lot of different teams and any improvement here can really help speed up the process. So it's really I think it's really worth focusing on the process of specking and the infer that we have for those specs in order to accelerate Ethereum's development. And if you look in the context of Par's talk, all the effort that goes into testing like after specs like everything they do, ctorsis, attacknet,

the hive infrastructure, shadow forks, everything, if you can catch a bug in an EIP in the spec, like just some edge case that we didn't think about, if you can catch that really early, then you're going to save a bunch of time downstream. And that's really important. And so we can really accelerate Ethereum forks. So I'm going to motivate our proposal of how we couple these things by having a look at the history of specs and testing on the execution layer which is the area that I'm focused on. And um first of all we're going to look at the genesis. So at the beginning um specs were basically in startup mode. So we had Ethereum that's a great thing. we went from zero to one and we were

testing evolving decentralized infra which is a nice problem to have. Um so if you look on the right hand side you've got um the different sort of repositories and things which represent the objects we need in order to do this testing. We've got the test cases at the top which are originally in JSON and YAML. That was how they were expressed as source. Um, and then you've got the test framework and you've got the IPs repository which serves as as the source of truth for specs. Then you have the client teams below. Can probably add a couple there later on in that era, but um they somehow were the spec. Maybe it was a bit unclear. As you can notice,

there's no spec on the right hand side. And um before I go to the pain points, I just want to give a big shout out to Dmitri who really like generated created and maintained a huge corpus of tests and Ethereum tests and these test cases are still used today. We generate them with our with our infrastructure. But despite that, we're in startup mode and there are some pain points. The test cases are quite hard to read and the tests were not parameterizable. So if they're not generated programmatically, you really literally have to write a new test every time.

In the next era, which starts around 2022, um when the EF um created a testing team, which I became part of, and the first step we did was we took a prototype from light client and then built it out into new testings for infrastructure that we use today. And this made the tests executable code. This really helped improve a situation because we had more readable tests. You could read them more easily. It made them easily parameterizable and that really led to faster test implementation. So as you can see on the right hand side now we've got the test cases in Python. Up there should say test infrastructure too. It's all bundled in the same repository which is

quite nice. It's versioned together. Um but the IPs remain on the side. And the pain points in this period were were more pressing than than the maybe what we're going to talk about today. But one of the clients acted us for spec, which is like a bit weird. Don't really want to do that. That's what we're testing. Um, and testers struggle to track MD spec changes. So we basically had to you know keep an eye on the EIP's repository and make sure that you know any change that came in fixed typo is it relevant what's going on like keep everything make sure what like version of a spec are we on it's not version just a commit hash what are we trying to spec here and then worse than that we really scrambled to find a spec

so like we really had to like go into someone's branch of some client implementation and find whether like it was up to date with the spec that was on EIPs and get everyone to talk the same language and make sure we're all the same commit hash, which is a bit of a mess. Um, and so that was a situation during the first era, but nonetheless, it was a step in the right direction. And in the second era, we another project that was being built out in the on the side execution specs, which is Python's uh sorry, it's a Python implementation of the of the execution layer spec as executable code.

And this suddenly appears as an object on the right hand side and luckily the clients can really become a neutral entity and they actually get tested by an independent unambiguous spec. So that really was a big improvement and it enabled faster spec prototyping. But there was still one big pain point and that was the tests and the python the specs sorry the specs and the tests were in two separate repositories and they would often get out of sync and that was a bit of a pain. That brings us to the era we're in today, which um was only about a month old. Um so we made a bit of a push in the last few months to bring all the tests and all the test infrastructure

over to the specs repository. So we we merged all the test infrastructure into the into the execution specs repo. So we're one entity. The specs have a nice home with its test together. We couldn't call it the uh the merge. So, we called it because we're steel the weld. And this was a really nice this is a really nice improvement because um the changes to spec and tests are atomic. So, if you make a change to the spec, you see the consequences downstream in the tests and you really see what the problems are going to be, how what's the workload. It might be it might seem insignificant in the markdown of an EIP but when you actually go and update that in the infra

you you potentially have like a huge amount of work to do. So this makes the test that tests are no longer some dark corner of core development. They're really right there next to the specs. It's a big improvement. But are we finished? Well Rahul and I think we're not quite finished. We've got um another suggestion today to try and relieve ourselves of these remaining pain points and they're they're quite significant. So at the moment we kind of have two sources of truth. We have the EIP spec as markdown and we have EIP spec as executable code. And there's no tight coupling between these two things.

They and they often diverge. Um, and this means because updates happen at different times, they're not in lock step. Um, a lot of the discussion happens away from where they should really happen, which is on a PR to the spec code where things can be disambiguated. And I mean, of course, we don't want to stop discussions elsewhere, but it would be nice if we have one focused effort to make an atomic change on the spec in both pl both places.

and I'll hand you over to Rahul who's going to explain this process and what our proposal is for this. >> Thanks Dan. Um so whatever Dan spoke about today um like the effort that we've done so far has been in pursuit of saving time. um having like test vectors, having um an executable reference and then having um everything at one place is an effort to save time and these are powerful tools but there's a limit to which tools alone can help us um and we must bring people together to to u save us uh even more time. So today we lose quite a lot of time um to these

questions unfortunately and um and like we have a proposal to like look into this. So um if you take this example Tony who is an EIP author has u has a question um and then Guru comes in with a valuable insight that eventually made um to the EIP. So, we'd like more of this, but quicker and closer to the EIP process. Um, and so we're doing this by adding a new role called technical reviewers to the EIP process. Um, so your role as a technical reviewer is not just to read text like

everyone else in the community, but you bring your deep technical expertise to convert English to code. um meaning that you help write uh the executable text test uh specs and write test vectors for um a single change. Um and I'm excited to show you how that works. So let's say if you are an EIP author making a change in an EIP repository um each EIP will be assigned uh a technical reviewer. So whenever you make a change, these are GitHub code owners who will be pinged when you make a change. they will take action to execute that inspec and they will also write test vectors here. Um

and then they will create a pre-release version for for clients and clients have an opportunity to review it. If they have feedback that can go in here in the cycle and if not the end of a change results in a test release and only after a test release the EIP uh PR is approved and this test release is encoded into the EIP review process. Um, and TLDDR is in goes a EIP change, out comes a feedback loop, uh, followed by a test release.

Um, so it might appear a bit uh, paradoxical that we're doing a lot of effort and making the PR itself complicated, but we're already doing this in in fragmented efforts um, throughout the cycle. But I uh but we believe that if we do that uh in a focused effort by bringing all the stakeholders together up front, we can iron out the smaller issues in the IP and help uh make the EIP a little more polished. Um and there for years there's been a debate whether the spec is the source of truth or whether the test is. Um today we can end that debate. Um and we can bring back um like the the test release

vectors as a artifact of the EIP itself. Um and by design they're in sync. So, so if always a change is encoded into a test release, uh you don't have to figure out. So whether you speak English or you speak code, you can always bookmark this page and you can be sure that you know you're on top of uh uh the latest of any and you get versions for free. Um so uh we're excited that you know we to bring this new role to the AI review process and we hope that finally all the Ethereum changes are atomically bounded into one package that contains both the

pros and the text and the test accompany them and we hope this creates a faster release cycle. Yeah. >> All right. So, uh, basically Rahul explained how we want to tightly couple this and we s almost have everything we need to do that today. We have, um, the specs and the tests together in the execution specs repo. So, we have reinfer on the infra side. We're complete and ready for this. And very soon to be merged is Alex and Tim's uh network upgrade inclusion stages EIP which really explicitly states um for scheduled for inclusion for SFI

um you really h must have a Python implementation with tests um and have that pointing to execution specs. So we must have an executable ST in order to go to the next stage. And just to make it clear um as Jiune uh pointed like described the journey of an EIP before so nicely this is really not for early stage EIPs. This is for EIPs that are really considered for inclusion on mainet. So how would this look just to summarize quickly um using our nice boxes on the right hand side. So as you can see on the right hand side if we make the small change we get a tight coupling of specs and markdown. So we get the executable

specs and the markdown specs tightly coupled through a process. So the improvement really means that um someone from the spec team or someone else who's qualified to do this work, it can be someone else um can evaluate the spec changes at like at the source of a change in the EMP's repository. So when a a change is triggered, it goes through the whole cycle of being checked through the spec. Um, as Rul mentioned, it's really nice because, um, the EIP's page on the website that everyone's used to, that everyone goes to to check what the status is, that can still be the source of truth, and it links back to a tagged version of the spec in execution specs

or consensus specs so that you can really see what the specs look like at that version of the document. And basically, as a rule said, we're just here to try and save time. So this can lead to faster hardening of EIPs and save all that downstream testing effort, all that effort of clients making climate implementations for specs that are changing. Try and target and get to the final product a bit more quickly. So of course nothing's perfect. Um but you know we have to find a balance between process and flexibility. I mean we think that most issues can be solved by social consensus. Um which is basically the situation now. So that's fine. um it's not too intrusive on the EIP process, so we don't need to mess

with governance and and uh have too many rules set in stone. We we think it's flexible enough. So, thanks a lot for coming and uh yeah.