All Core Devs - Consensus #137

Transcript

for [Music] time [Applause]

Alex Stokes: okay I think we're good there and here's the agenda yeah issue yeah okay great to everyone this is a cons Slayer call 137 it is issue 1096 in the PM repo and yeah let's go ahead and get started the agenda is pretty light today so so hopefully uh yeah we can get through everything so first off I want to start with Electra uh in particular devnet

1 I think most CL clients are ready for devnet 1 um does that sound correct everyone you need some thumbs up okay I'll assume your thumbs up mean means your client is ready to go so then otherwise I think last time we were waiting on some e clients to be ready um that being said I think there are a couple that are ready for Dev net one so I would suggest we go ahead and launch the devet as soon as we can I think uh par and Barnabas are out

this week so they are usually the ones doing that but that being said I think we can uh launch it as soon as we're ready does anyone oppose that I think even if there are other e clients who aren't quite ready they can just join when they are okay there's a ball pit we can't hear you oh there's Barnabas

oh yeah PK

Barnabas Busa: hey maybe I can quickly join in uh we are currently testing with teu nimbos

loadstar and grandine as the consensus layer clients and nethermind and G as the execution layer clients and when looking at the uh interrupt chat I've just found an small issue with uh nimos that should be looked at but uh if any other client is ready please share your client image that we can add that to the testing apart from that all other clients are fine so far

okay great

Alex Stokes: sounds good um cool then yeah I mean I would suggest we start DEET one next week if uh yeah that's not possible then I think definitely bu the well so

be the next ACD next week yeah I think as soon as possible and uh yeah we'll key pcture moving along oh one second um okay yeah anything else uh on Petra or Dev net one that we need to discuss right now

nope

okay um great

Alex Stokes: in that case uh we will turn to P dos so I don't know if anyone here is on the breakout call the other day um but it sounds like things are moving along I think there were some issues with the devet that they had um but the plan is to move into the next dnet anyone have any peros updates they would like to discuss yeah so we have P DEET one running now and it's currently forking away from every every client from every other client so we really don't have a good uh DEET one we were thinking about

a relaunch and on Tuesday we discussed that we are going to be planning for a relaunch by the end of the week so hope hopefully by tomorrow we should have uh client fixes for all the different issues that have arise on Def One okay great cool so otherwise uh I have a small update uh I had a PR to start to uncouple having the Bob counts set on the CL and

elel uh after some discussion there uh what we landed on was that essentially the check even right now for the maximum in the El's redundant uh just given the way the engine API works and the check of SE already does so the plan there is to actually keep the maximum check just with the CL uh that does not need to be sent across the E because it kind of already happens indirectly with the Virgin hashes there's in the question of the target the target if we want to have the CL sort of drive this value will need to be sent across engine API in the same way and uh also be included in the block

header so that will look like an EIP um that presumably we would include inetra or sometime in the future and yeah I have had not I have not had time to get that together but uh yeah just a quick update there that's where things are at

Mikhail Kalinin: um I have a question on that topic um related to this and uh this uh PR and 10 CR PR um could you please give us quick summary on why can't we um compute the Bob gas price on the CL and do the validation and give and send this value

to to to the El right because yeah because in this case uh we would not need to add another field to to the block header as far as I believe uh we would just could just repurpose the excess uh field to to keep this um price I don't know if it work

but yeah

Alex Stokes: yeah I know that that sounds right I mean the main push back with so there is uh this PR that Shia dropped uh it's can sucs issue or PR 3800 so this one is just basically handling like these Max and Target values leaving all of the fee accounting and calculations with the E itself there

was another PR from Den crad to basically even go one step further and pull the fee calculations into the CL as well uh we can do this uh I think it violates the sort of you know separation of concerns across the E and seal that we have and generally is nice so from what I heard is that uh implementers with pios kind of preferred the Max and Target routes rather than hoting everything into the se but it's an option

Mikhail Kalinin: I'm just trying to understand where is the separation of concerns would be break broken in this case because it's more like the you know

I it

Alex Stokes: just seems to make more sense to have

like fees and accounting and all of this within the El because that's like like the CL doesn't really care about like the blob Bas fee for example whereas the is quite a bit

Mikhail Kalinin: Yeah but I would argue that actually C CL cares because um CL handles the networking for blobs and blob propagation so from that perspective C care about this Target and this is why actually um we are uh Our intention is to uh have these marks and Target um given by the CL rather than defined on the El uh and also yeah but from the transaction propagation perspective um here is just yeah just

one could be one problem but I don't know how how much uh this price uh is used for in The Blob transaction mle the probably is important to have uh this this value at hand to build a block and propagate transactions in this case yes it's it must be on the L as well known in advance

Alex Stokes: right I mean so the definitely needs the target so it can compute the base fee uh the base fee is helpful in the M Pool and block validation all these different places I would say the CL actually doesn't care about the target it only cares about the max um again just even more for like dos

concern reasons

Mikhail Kalinin: yeah I see okay cool thank you

Alex Stokes: that's how I'm thinking about it so uh I'll keep working on the cons specs PR and also any IP and and uh yeah we will keep those moving along if other people feel strongly about you know this other option of having the the base Fe calculation pulled all the way into CL like we can explore that as well um generally from

what I've heard again we want to leave that with e okay anything else on

pios okay uh then

Alex Stokes: next up we had an update on Fork Choice testing uh let's see is Erikson 49 that's the get up handle are they on the

call I don't yeah

yes I am on the call okay perfect

Alex Stokes: you had a presentation

yes uh I put the link in into chat oh can you see my screen yep okay great uh so we have been working on for Choice compliance testing uh it's a work supported by AUM Foundation Grant actually and okay so we have implemented

test generator and generated test s there is a link to a more detailed report on E research and I quickly go overview of what have been done so actually the there is more work but we have implementing implemented initial phase uh this generator and the compliance test feeds and our Focus was on ease of adoption minimize surprise to client teams and in particular we keep the same for choice test format without actually there is only one small changes we added

new check which can be ignored initially and Next Step will be to uh work on implementation specific coverage guided fuzing and using this test suits as SE Corpus and eventually we expect up to 1 billion of test vectors but it's probably up bound okay so test generator is implemented now we have a draft consensus PC request to put it into the repper uh okay as I said it outputs test

cases using standard test format uh we have also simple python test Runner which can execute the tests using the official work Choice specs um the test generation is parameterized so there are several parameters and we can adjust them and generate more test depending on your needs uh it's currently one problem is that test generation currently is slow about 10 seconds per test we know the problem it's basically a consequence of consequence of compatibility so we are relying on the existing for Choice

infrastructure and because of that one aspect is that it's kind of slow however it can be run in a multiprocessing mode okay I hand over to M to talk about

Mikhail Kalinin: um thanks Alex so this slide elaborates on the E part that Alex has mentioned so we have added just one more uh check or the for into the FK choice test format so the the test generator produces the um FK choice test in the usual format and uh yeah with just one additional property which is called viable for head roots and weights um and

uh yeah it can be ignored at the beginning so uh it just can be skipped uh this check can be skipped um yeah um so that's that's the extension of the format so it should be pretty easy to um integrate this one um next slide please yeah so now let's a bit Circle back and a bit describe how this test generation seat Works uh so we have uh a um three actually three um models written in minut sync which is the constrainted um Sol solver language uh this is one of the models it it gives us

an idea aidea of uh why why do we use it and how it works I mean the constraint solver in this particular case so uh this block cover model uh covers the various uh sets of predicates um that are in the filter block tree to define whether the the block is viable for hat so on the right we we see the exerpt from this spe so the predicates are uh kind of uh Flags which indicates whether the ENT Source Epoch is equal to the Justified checkpoint Epoch whether it is uh whether this Value Plus two uh is greater than the current ook greater or equal to the current ook so we we

actually give a bunch of uh a different set of those predicates one can be false the other true and all this kind of stuff uh so we we give this Vector of predicates to this model as the user input to this model in this model reduces the solution which is written on the bottom so this just numbers uh and some you know yeah some information that will be used by test instantiator Downstream uh which will produce the test uh the FTR test out of this um yeah data and and some other um yeah so the here what you see is just one solution but we can ask constraint

solver to give us all potential solution all possible solutions uh with the certain constraints uh on the next slide yeah we have a couple of uh other models which uh used to produce similar information for super majority link trees and for Block trees um yeah and and the next slide please so the test config looks like this um there are instances instances are those solutions from the model from the given by the constraint solver that we have already uh seen uh there is also the randomness seed uh that is passed on

the test um on the test instance input and the number of variations and number of mutations uh number of variations is the um number of variations with different with different uh Randomness seeds uh and this just the initial one so the f one will be derived from from this one um yeah next slide please so how the BL Randomness uh seed is used uh so this kind of like the block three instantiator that uh consumes the super majority in block three um model outputs and some Randomness seed and for instance in this instant here uh we take this data and uh

we use the we leverage on the existing Fork choice test helpers from the ppack test Su and produce the um fully functioning um fully featured for test and uh uh just wanted to mention how this uh uh just want to show how this Randomness is used here uh uh so this particular instantiator Works in two uh steps first step is to create the um three of super majority links um given some certain um model constraints and uh the Second Step would be to create a to to build a block tree uh based on the most uh recent Justified checkpoint U so

and on every on these two steps we use Randomness to actually partition validator sets uh and sub sample validat sets that will be building the blockchain uh during the certain period of time for instance during step one we create partitions uh validator partitions um uh to to to to be active during one to be active During certain certain Epoch so we just yeah use Randomness to sample um one or more validator set subsets uh so it can be two as you can see in the third ebook uh we use to Val subset because we need to build two um

parallel um two forks two parallel Forks um yeah and yeah the next on the next step this same Randomness used to attach to different block tips and also used to flip the also flip the coin when we need to whether this slot will be empty or not and for other aspects of uh the test as well but yeah this gives some uh intuition behind how Randomness used and so the with the given super majority link and block tree uh outputs from the model we can apply different seeds and and actually get get different test cases with different FK Choice uh with different weights for

each uh viable hat and uh yeah different uh and some other properties like empty slots and invalid messages and for so forth next slide please now to the test seits Alex do you want to give

yeah

yes yes I just switching on my mhone okay so um again more details can be found in the link uh we currently implemented uh three test Suites like tiny small and standard tiny is like for demonstration

purposes and small is for initial adoption and probably for smoke testing and so and standard is like for main testing and yeah okay standard is 13,000 and small is 10 times less and so on we also planed to generate extended test suits however currently since test generation is low and there is no kind of demand for it we did not generate it but we can do that in future if needed uh there are also links to test on in t format

links to the generated tests and yeah okay so a couple of wordss about test groups uh we split all test in six groups uh block three it does Main testing kind of focus on trees block trees of varing shapes and block weight is more focused on producing block trees with a variation in weights to cover get weight functionality and also have a shuffling group uh which focus on shuffling mutation operators and and shuffling basically means that we uh shift uh events kind of delay

block or make it appear later or earlier or drop and duplicate can also duplicate messages also have a tester slashing uh group of tests and invalid messages yeah and bl cover which covers various combination of predicates from Filter block tream method yeah I hand over to M again

Mikhail Kalinin: yeah so we also did with this test suit we also did uh integrate it in teu so yeah there were there was a question about how fast is to run those tests so it's

like the standard Suite with the 13,000 test takes about 20 minutes on on my old Mac uh to run which is yeah quite fast uh so we have found several issues of different kind uh like were two HK bags in the test session testation processing in teu um so yeah but they are Super H edgy and yeah unlikely to happen ever but still uh yeah what uh what interesting uh is the next one is that uh there were a couple of issues a couple of fail tests uh that are related to to um attestation process and

optimizations so this spe does not nothing about optimizing attestation processing but clients do those optimization to prevent spam um on the main net uh we had the incident uh a few month ago several months ago so uh and yeah so actually some of those attestations were not processed by TECO which is fair those optimizations are pretty leged uh but this is uh this caused the Divergence uh from the test result from what what's given by the pack and what what happened in Te also uh in Te FK choice test executor deferred uh attestations from the future that is coming from the future for later

processing and applies them later on it's kind of happen implicitly but it does not defer blocks um and to say that on the main net run it would defer blocks and process them as well but the FK choice test executor um triggers uh those parts of the functionality with t uh that would not defer blocks so we we have this kind of to deal with this somehow also we we ran those that test suit with the coverage was just stand standard coverage U um metric um and yeah the following functionality

appeared to be not covered by the suit um so obviously with that that was expected that execution pay lot invalid status and all all that is related to it um kind of like remains uncovered because we do not model this in our test uh seit and also the were Pro array pruning uh threshold is quite large uh to be hit by our test because we run we produce those tests with minimal configuration so it would require like main net configuration to um to hit this uh this um yeah functionality and yeah on the next slide

we kind of like uh summarize an open questions uh to to all client developers um based on our experience with teu so how's how how to deal with the first question is how to deal with the messages from the future so we have several options here and ideally we would like to um yeah if all clients uh defer those messages for later processing uh then probably this is the way to go so the suite can do this uh and expect that the client will implicitly apply uh the corresponding message uh when the um when the store time allows to do this um so that that

would be the best option uh I guess but there are others to think about and I believe it's pretty much dependent on uh the folk Choice uh test Runner implementation in each client and on the FK Choice implementation at all so the other one is how to deal with legit optimization and client implementations uh so option there are two options here so the um the first one is to just ignore those tests by each client implementation separately the other one is just exclude them from the suit somehow but I honestly we don't like the second one because different clients may have different optimizations and uh some will work uh some will not

work so excluding them all would be kind of like meaning that we not test some functionality in some client because the other client does not process the attestation for yeah leged reasons so those are open questions to think about uh when you will yeah when this uh test will be being adopted next slide please yeah so basically that's that's all we have uh so far and happy to address questions

Alex Stokes: super cool uh I have a quick question like how hard is it or easy is

it to uh write these constraint models like did you find that part was like pretty straightforward

this question to Alex I believe uh I

it's probably writing models not very difficult but um making them useful to generate tests can be tricky actually we originally thought about using some kind of test generator would just give us kind of best coverage of State using standard coverage metrics like Branch or statement coverage but in in case of our choice

it's not very interesting actually I mean it's easy to get and so we come up with this solution to use a modelbased coverage so for example we have several predicates in uh filter block Tre and we want to cover various possible assignments of true and false values to these predicates and of course some of them are not probably possible they can be excluded by some checks in code but we want basically to have a test case for each combin combination of predicate which is possible to instant

shap and similarly for blocks and for trees we can say restrict us to have like say a trees of eight blocks and just enumerate all possible combinations of such trees probably with additional restrictions like maximum child count and so on and so this models we use these models basically to instantiate um models okay instan shate this uh enumerate all possible com variance solutions to this model and basically when we

generate a test case for each instance this will give us coverage that we want and writing I mean it's not difficult just we need to have some experience with constraint programming constraint satisfaction programming it's actually quite convenient I would say

gotcha thanks Anar Inus not not Anar here or Anar

Ansgar Dietrichs: with a different German accent um uh so regarding the uh letting people uh or forcing people to to to

um uh forcing the test to to leave our tests that people cannot pass so we kind of have a similar problem with the with the execution uh layer tests where we have some tests that we cannot pass anymore because we uh for example we removed the um the for choicer um that was needed uh for Prem merge so all of the tests that kind of require you to um uh to pick between different proof of work chains uh are stuff that we cannot test anymore and we just um ignore them so I would I would also go the route of uh telling clients like this is the test suit and

if you if there are tests in it that you cannot pass because of your optimizations you should just uh ignore them locally um for your client but not remove them from the

tested

yeah sure thanks definit there will be some uh kind of cases where clients do not pass test but they does not mean that this is dangerous because for example if some message is early delayed and but there are multiple copies of blocks and adust stations arriving in practice so I think many failures won't be real problems and we

need somehow exclude them

definitely whats

Potuz: I'm just wondering how hard is it to change the St generator for other versions of for choice I particularly have in mind uh 7732 but also for pier do if they eventually go to block slot voting how hard would it be to adapt it

uh okay we can generate basically we can we are currently generating for the nap and but we also try generating to Al and capella and using main net and okay

we're generating using minimal uh profile but generating main is possible is just very slow and for other Forks basically it should be working hopefully it will work just need to

Potuz: so this these are Forks that change the way that we that change the for Choice logic actually is not it's not about main it or minimal or not the the form of the blocks but actually the the way we count ATT stations on the way we wait the nodes

yeah yeah I I mean it still probably work because we just generate this blocks and

trees and just can run the same actions so I I kind of don't know details maybe it will require some adaptation in in a good if you're happy if you're lucky it just works it will just yeah

I

Mikhail Kalinin: I can add more on this um probably we'll try um so yeah as Alex said it might need to even uh yeah some FK Choice changes some f Choice surgery might might require um adjusting those models those constraint solware models that we were mentioning in the beginning uh but for this uh block slot particular

case uh the empty slots are kind of the product of the randomness seat that is given to the generator so it probably will be enough just you know to to do nothing and this will probably be enough uh having the empty slots yeah

Potuz: but M that's that's very hard to believe right so on on the Block slot the the weight would be assigned to a different to a different node right when you're asking for weights on one of these optional tests I mean the result would be very different on one model or the other model yeah a matter of like changing

yeah it

Mikhail Kalinin: it's not going to be a matter of changing a test generator because test generator uses this pack

actually so it directly calls to this pack if if this pack has this change if the spec has the uh block slot for choice in it uh then the generator will just use it and produce the outputs correct uh compliant with

this p Ah very nice

Potuz: very nice I think I got it thanks

yeah but

Mikhail Kalinin: if there is some change that requires to remove some of the predicates or add new predicates uh then we would need to yeah that's just an example then we would need to uh adjust the model that's yeah works with and probably to improve Pro coverage

you also uh may want to introduce changes

y so

Mikhail Kalinin: are there any other questions

okay if there will be questions you can write us

Mikhail Kalinin: yeah just ask us directly in Discord whenever and also we're keen to see this adopted by um client different client implementations uh we have a PR in teu um the the pr is listed in the references slide uh to this presentation so also those test suits are public and we have this PR to

the consent spec specs repo so please yeah find a time to to to try it out to run it so the presentation slides all yep thanks everyone for the for your attention

thank you

Alex Stokes: yeah thank you both

super cool okay that was everything on the agenda is there anything else we would like to discuss otherwise we can wrap up early today okay

let's go ahead and call it thanks everyone I'll see you next time thanks byebye thanks thanks thanks bye

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.