EOF Implementers Call #55

Transcript

like to welcome everyone to eof implementor call number 55 on the 7th of August 2024 um lot of agenda items relating to testing we'll get there when we get there um one is a spec question but the rest is mostly testing um but we'll go ahead and start with client updates um Basu implemented uh fuzzer over the past few weeks it was uh Guided by Java coverage and it's differential against the others and um I've had good overnight runs with evm one and W with no findings between the three of us and I found some items in gu and JavaScript so that's some of the stuff that bases

found in their development um next is uh evm one I think now updates on the implementation of vm1 we're we're we're kind of done but also maybe others can share some on fing as well and the rest of the team is also involved in testing Can you sh on cling part we cannot hear you at least I cannot yeah your microphone's not coming through P yeah it's like just noises pleas how about we we go to re and wait

for p to fix his microphone uh side P found with his fuzing some bugs that on verification that was fixed and there was some unimplemented things there uh that's mostly it I still need to need to figure out that Matrix and update the tests I I don't think I'm running East tests so I need to check that later today may maybe tomorrow okay um do you run your test by dropping the fixtures in the directory

and running it is that the main way you run rehab a tool that you can point it a file uh I'm pointing it to folder basically I basically downloaded Mario bundle of U tests and I'm running those okay because I have some automation for the ones that'll run it from a CI Tool uh but I haven't updated much of my of my automation it wasn't very wasn't very expensive um any luck on Peter's audio situation right you me Pavo correct

yeah Paul not Peter wasn't he the one that was going to talk about oh he's yeah yeah exactly I don't know you're on mute P not sure is it good yeah now perfect great you're gonna talk about evm one fuzzing yes it's broken again such bad luck so is the fuzzing that the evm one runs is that checked in any directory anywhere that people can run I don't know

okay um how's microphone is still very unhappy we we need mic Dow we need a need microphone Guild I guess oh he he wrote in chat oh actually okay so there's the fuzzers that he runs uh it is on UF Branch yeah and this is just for the E validation right correct okay because we got to validate which is the newest this is probably we'll save this for testing actually um I don't see get on the call I don't see NE mind on

the call I don't see ethereum js on the call um ethereum JS do have a couple of findings from the Java fuzzer um but they don't seem like you know they seem like the the findings that other clients have had there nothing out of the ordinary nothing truly bizarre um Guth has has more findings I've um fix some in a PR that I'm going to push to Marius when he's back from vacation he's on vacation this week but I haven't fixed all of them I've been building other things um how do you think you got your mic fixed now I can

try this is this one working this one is working yeah and hopefully when I close the settings on Zoom it will still work don't close them okay can you hear me yes tell us tell us about the the evm one fuzzer yes so it's similar to to Bess except the coverage is taken from evm one uh RM and and Bess is integrated on the side so we don't have coverage but we can we can ask it for uh for validation

um so I think we I I I can confirm Bess RM and tm1 are in sync in this and um I need to update the mutator for for the fing because it's pretty bad and uh yeah and I think that's that's the story so far uh about integrating others uh we can I think we can do a go code with coverage and the others probably without coverage um but I think I will put the uh a a baseline that the other projects

needs to pass the the existing tests uh so at least have confirmation they they are there because I think we we we know that that the g code is not great um yeah yeah so I'm sure this really benefits um yeah but that definitely can be done um also there's option to export all this uh kind of generated tests into Json not sure this is worth doing this but uh that can be also kind of the one one more like gate to cross

uh to be included but I plan to include everything and and I honestly think any good buzzing find uh probably indicates a testing uh blind spot um and it's usually the the combinations you don't think of like um tripping the reverse jump and the forward jump with two our jumpies uh two conditional jumps in between that's that's one thing that got Basu and helped us find uh implementation thing that we weren't doing correctly um but yeah think any any good fuzzing find usually should be followed on with something added to the reference tests um because if one client coded it wrong the odds are other clients or future clients will make the same mistake yeah um there's also this uh

list of issue found so far so it's kind of work in progress so so far we are collecting these as the someone remember the link it's it's on uh ipsilon e actually um so they need to be processed later uh and submitted to execution spec test yeah so this is yeah good good fuzzing findings are being found here so um let's see compiler updates back updates let's do some of this fuzzing stuff first before we go too deep um while we're on it um so if we find

fuzzing stuff we should post it here but that also gets to some of the um um standards I don't know the phrase the way to phrase it but when we find um bugs in pre-production code it's generally fine security security um uh there's a security aspect for it but if it's pre-production code we can post it here but if we find a bug in production code um we should probably have a a narrower disclosure um for operational security um so if find a bug that is live in in gu or Basu or or nethermind or evm one um that could be used on older versions of the code that some of the stuff might show up as we do the

fuzzing we need to have a narrower scope and I think there's a keybase disclosure or we can disclose it privately but when we're talking about code that we haven't shipped to main net and that's a new feature um I don't think we have the same risk of just disclosing all the bugs that we find So eventually this this will shift we will need to um have more limited disclosure on some of these but for now um we can just disclose freely anything that is an e e eof specific bug and so that's what these will be filling up um okay so I think that's all the client seems go ahead there's one more

thing about fuzing so this is the issue that I'm hitting uh probably with every project is that uh my expectation is that we we kind of uh validate in runtime mode um I think that comes from the fact the rtime containers are simpler and that's also how um the validation test are organized my to my understanding but I think it's maybe that's actually the bad decision to do it this way because that's actually not something we will have um in a client because the client will always validate in in it code because

that's because we we only validate the the eof transactions right so we always kind of validate uh top level starting from init code also to validate we also need to validate Genesis files so if you get a Genesis file with an existing eof contract we need a valida part that as well I believe yes it's true but maybe for fing it's actually better to switch do you need code I'm sure if you have opinion uh so far this this fixed but I I I had to kind of yeah kind of work around it a bit for both peso and RM and we kind of hitting some a bit more complicated entry point for that so I'm not sure that's that's a

good idea but uh yeah so I I don't know if if you have any suggestions I was thinking maybe there's also way to to actually combine this two somehow uh let's say you passive input for validation and you get instead of specifying what the what the type is you get this as the answer and then you can compare if they agree or it's not like exactly how evm is is implemented but evm one is implemented but uh yeah I have this issue probably I will not do any actively try to to to change it but I wanted to mention that and if you have strong

opinions that we shouldn't we should actually change the the validation mode mode by default for fuzzing I can also do this we should probably fzz both input mode and non-input mode non code mode um but that creates more it creates a difficulties in fuzzing because there's a lot more different ways um that you can be invalid with a nit code and and bring everyone into um init code failure um so that's yeah that's that's something we'll need to to think about over the week um I can add a flag to uh bases code validate to expect a nit code and I

can do some runs where we start from a nit code getting the seed getting getting initial Corpus for nit codes is going to be what's difficult um but that should be fine we should be able to figure that out yeah there there's at least a flag in the ston test that specifies this and this run time by default but you can specify it h also for besso I kind of fix that by by just yeah just adding some new lines to Java but I think that's done so it's not something I need directly right now m but it's kind of repeating pattern

that uh when I try to find the entry point for validation in other projects it's it's a bit difficult to specify it sometimes and I kind of want to make sure that we fast something that is then directly used in a like bigger client implementation not something that is kind of internal API to something that it's not used so like a transaction fuzzer we want to spam the transactions Mario yeah this isn't related to fuzing but I wanted to mention uh that we also need to generate status for the container runtime validation test that we have basically just wrapping the

runtime uh container into an init code and then use that as the transaction Contra creation uh data um just want to mention that we don't we don't want to miss this uh this coverage because yeah as you say uh most of the container validation in in actual in an actual chain is going to happen during init code um so yeah I think we should do we should do this it's in my to-do list I just haven't gotten the time to do it yet cool um so let's uh table testing for now come back to it further down the

discussion um do compiler and spec updates I don't see any compiler people online um there's no update to the solidity PC I take it I know is working on on it from Epsilon there's some some progress but but not here okay um the the the situation for solidity is that uh the implementation has been kind of submitted as a PR request and this under review and uh yeah so that's the main main thing there are some optimizations probably a lot of optimization that has

been disabled because UF so they they also work on the on a way to reenable some optimizations y that had to be disabled because they're not like directly compatible with UF and um I think and finally we we we we want to make a evm One release and evmc release uh so that solidity can update this dependency that they use for testing and then that will be like separate work that we also want to we kind of need to to put in between the E is merge but uh yeah I think that the end goal is to actually merge this Cod

to the to the main runch okay okay um so I think next we have

spec and there's a couple of things um first we need to figure out um contract detection and then I think I had on there um some tracing changes so first contract detection um Alex isn't here yet but he did post some stuff to uh Discord um basically we got you know six options to go around it and there doesn't seem to be much excitement around some of the more um larger changes like multiple return codes multiple return items on the stack um features like this um and really what

a lot of the interest is around I a completely punting for the first release of e and coming back later um or B um using reintroducing um EXT code size or C changing EXT code size in something like e um is contract um yeah any any comments on on the state of that do we want to fix on a design or honestly a lot of the uh the speed we're going the uh the do nothing inertia is getting a lot of weight behind it well they do nothing the that's the big advantage that we can fix forward fix

it right right we can fix it with a compatible change in the hard Fork after eof we can do a better time evaluating these options prototyping them seeing how they would work because if we bring any of the other fixes in we have to live with them forever um even if they're the right fix um that would be fine but if we bring in the wrong fix then it's it's it's a change it can't be undone the um not supporting 721 and 1155 contracts on the first pass is the most undoable of the options and it's also got the potential of being the correct option um if account abstraction really gains traction but that would basically preclude eoas um from Holdings some 721

and um 1155 contracts but um you know that might be acceptable if if AA is really going to take off I me to to comment on that 721 and 1155 right uh they can still I mean we're not making them impossible we're just either they need to be written in Legacy evm and deploy as such um or they need to call out into a legacy contract which just a uh EXT code size or whatever they need for them it's it's um it's still possible they just need to pay I don't

know um extra right or am I wrong on this hello hello I'm writing this down again you're R it down okay yeah oh yeah that's a spre option right they can um somehow not use the safe trans transfer from right but I guess when we are sticking to principles then

uh I mean either a contract satisfies the the interface the the standard or not um all the calls would just fail I mean yeah I mean the interface 721 has got a lot of optional calls but all the major markets require save transfer from if you want to make your nfte marketable so it's a practically requirement we can't do um if we don't do the return code changes now we can't

do them in the future but Asic and Discord said that the Unix standard is zero means success and anything non zero means failure and there's really no distinguishment between success types so we would move contract we would move transferring money to an EA into a a failure mode if we did that from from standard but I think the return code changes are um what had a lot of

resistance um I do think we should probably just stick with the do nothing for now since it's something we can fix later with a with the lead of the others I think is contract or is or EXT code size was something we could bring back into the future um so I'm fairly comfortable just sticking with the spec we have unless there's any any call for action on that now whether that's going to be in Osaka or Amsterdam remains to be seen um if you called in all cour devs a couple weeks ago uh or last week um there's

some concern about postquantum cryptography in veral so that might be shaking up the schedule so anything in the future there is no promise that'll ever show up and there's also no promise that it won't show up so it's just not on there okay um moving on from so contract detection I think the soft answer is sounds funny okay so next thing talk about is

tracing changes um and this is going to become kind of an issue when we go into um execution fuzzing um there's a couple of features that we're adding in eof that aren't this is the EIP uh what is the um there's a standard tracing API that most clients follow 3155 um and this specifies um most most clients spit Cod out like this and this is what a standard Trace looks like um and this is

what a lot of the the tra the fuzz the fuzzer fuzz off of but we're adding a couple of features that I think we need to consider updating so right now it's actually stagnant it's not final um probably if I do a change of this I'll change this to a living instead of a draft um but there's a couple of fields that I think we need to consider adding to this Trace um we're going to need code section and I think we're also going to need the immediate arguments um because I also know that tracing um some some ZK systems they arith matize the trace they don't actually run the evm someone else runs the evm produces the trace and they arithmetization to do ZK out of if they

continue to do tracing um is their solution they'll need this information to figure out um you know what not a push but like you know the call F what sections you were going with KF um Pua you raise your hand yes um can you please repeat the EAP number and is it actually being used by the client I don't like that it is an stagnant status I might want to check with the authors and push it towards you know whatever active status it should be so it's 3155 and I think the reason it got swept into stagnant is because it's not final um all of the clients have updated it and that to to various degrees and we have this and it's just I I bet you know

Martin um and Marius just you know they they they're not very attentive to process things on this so they didn't push it to final um but it is not it's not consensus critical I think is one aspect for it um you can you can have a consensus client or execution client that doesn't have this this is just a tool that is being used for standardizing testing sure I would try to get in touch with Martin and marus and try to move this proposal out of a stagnant if it is being used by most of the clients right what what what status would you recommend would living be appropriate for this um I think that would depend upon EIP editors given any of the interface

proposals are never been in list status I highly doubt that is possible but moving to a final status can be very much possible okay cuz the problem that I have with final status is is the changes we're discussing now where there's a new feature where we need to update it all of a sudden we're going to have to have a new EIP number for the tracing standard for something that's basically the same and I think this is like the poster child for Virgin eips um it is this tracing but then the the EIP editors are going to impose their will and say we I'm going to going around I should stop about I really don't like the way the E editors govern this so um I'll just stop there no I mean like the general solution right here is like if you think

that after it being final we might want to add one of the eips to the list there could be a new EIP with a required section as 3155 and the true line of change would be there and that can move faster because this has already been here I see the number 31 that means it is pretty old proposal and I can understand get teams are uh sometimes need some support to move the proposals to different status but I hope going forward there should not be a challenge moving proposals please reach out to me ping me or even join the editing office hour we will push the proposal as soon as possible so what's your recommendation should I add the eof stuff here or should I just do a new EIP for eof

tracing it would be a good idea to have a new proposal because I would want to see euf moving with the pectra upgrade I don't know how much time would this proposal take to you know move into the following directions I mean like the next status there are two ways to go about it either the initial authors have to open a new pull request I can make sure that they get merged as soon as possible but if either of these authors are not open to the idea we might want to add someone as a co-author you can be added as a co-author or Champion for the proposal and then when you create a poll request we will move it forward yeah thing is I I'm fairly certain Martin and Marys have given up on

eips oh if you would want to propose yourself as a champion or coauthor of a proposal the only requirement would be for them to approve you as a co-author and then from there you can take it forward and I'm happy to help push it to the next okay okay so we'll move the uh um we move this process disc discussion elsewhere um but what I want to discuss is get people's uh temperature for temperature check for some of the changes I want to put into this into the eof version um add an immediate that would post immediates this would show up for pushes call UPS anything that has an immediate um add a section number and

then um PC whether we do Zero from section or zero from container I want it that's the biggest open question I have there is what people think about that [Music] um Andre and dragon do you have much opinions or um Peter or Powell I'm thinking PC should be relative because PC together with the section are uniquely identified location otherwise it's a bit redundant if PC is absolute then section is kind of not needed yeah okay zero section okay um it it kind of depends like you want like efficiency or just

readability because like the same what Andre said kind of because the code sections are adjusted to each other it kind of the PC started from section one kind of works for all of them you just need to figure out which which section are in outside of the trace but maybe that's not desable I don't know um the usually this traces are not really efficient in terms of this rot of redundant information and sometimes it's kind to to actually have all of it so I guess that's that leads into my

next question is some of the musts um so whenever I do my my tracing and I test it out I actually only pick out um the PC um the gas cost the top item of the stack and uh the memory size um when I do my dipping because those are the most reliable ones that are consistent across clients um client some clients will do um basically used to just do not the uh for calls they used to do the gas that did not in call the include stien now we change that because lanaa needs it but gas cost I've been seeing and get spinning out two lines for the same op

code one static and one Dynamic for gas cost which is going to cause buzzing issues um and then yeah so um what required Fields so I one of the things I want to do is is make sure that we have gas cost have a specific meaning it'd be the gas dipen um gas okay gas must be gas available to the operation and the P section must be present in correct um so the gas that's available to your call must always be that value I think these are the two

musts we must put in for for tracing correct for using these for fuzzing any other people been using the traces and have opinions on it well debuging go La passing they they also had issue with gas so I think for some of the products they just ignore it because it's not consistent across these yeah yes cost yeah I don't know it's like always I I we we had number of discussions about it what exact meaning is and it's like for EVR for example it's

not always practical to to have the full gas cost and I mean it's doable I plan plan to actually like make it consistent and I I plan to compare it with G mostly uh what can be any other project to find differences and then fix that the the calls are kind of separate topic to that like what's it's like is the the gas put to the the Coler it's actually cost or not I think we kind of on the position that it's not because gas cost is something that is is kind of burned by the instruction so

it's not available to anything else it's just gone but I'm not sure this is consistent what what the other project actually put there and I think that's that's we can step by step fix this and that but it's kind of overwhelming process considering that goes to AP or something think okay yeah so I don't know if if you want this section section think I'm fine with that I probably put shorter names uh but that's that's the only comment I have uh there was a request from go evm

lab to have no memory and maybe even no stack no memory for sure because repeating memory in the traces in Json can be very performant very yeah it will take a lot lot a lot of time I'm not sure about no no memory for sure but no me for sure that's yeah I mean no stack maybe we could just have this top stack item yeah I was also like talking about it like if you only have the top stock item or even for only for the instructions to actually put new value on the stock you

can put one item and for the other se you can skip it yeah you can dve the tax date from there yeah you can kind of only having ajason you can figure it out what's what's going on so you can so like for example go you lab has this kind of debugging tool that you just take a trace and then you have the interface that you can pretend you're actually debugging existing program and you can step in step out and like step over different instruction um yeah so I think I'm open to also because no maybe it's enough to uh put like some number of top stack

items or we can decide there's like a marker that says there's more stack items but they are not just included in Jon okay so I'll drop a mock of this um we can discuss it online or in two weeks again I think we need to move on from this for time we got a few other issues um yeah so to drop a comment in this thread if you have anything else to add um let's go to testing Mario what's uh some of your testing what's the top of your mind for testing um okay so I think okay can we go to the checklist that you posted I think

there's some interesting oh yeah I I can address curtosis first so I think uh for the time being rosis I I wouldn't I wouldn't think that this is absolutely necessary um what we can do instead of curtosis is just using easts um consume engine uh uh tool basically what this does is it relies on hive but Hive sponds the client and basically we send every single test that we have using using this thing via the engine API um there's not much difference between this and cryosis what the difference being that cryosis does Spa the consensus client to do this but I

think it's too much overhead for the spot that we currently are in I don't think we're going to find any any significant differences between consum engine and kosis um the only thing that comes to mind that maybe CIS could uh maybe point out is that for example if we do call eof calls to the to the to the system contracts that are being introducing PR so for example the deposit contract or the um or what else the the withdrawal request contract uh maybe if the call comes from a an EF contract maybe there's there could be some some interesting uh uh some interesting things that could come up

when kosis spawns the consensus CLI but I I really don't think it's worth the effort of setting up kosis at the moment so yeah and coming back to the to the to the to consume engine I think consum engine does everything that we need so far and it's just a matter of running every single test that we have uh written for UF using consume engine um asserts also uh coming back to the uh to to to the to the the point of the East test that we have I think if we um could write some uh EIP interactions between for example the deposits EIP and EF which is really easy when since with

the markers that we included um in in I'll link the pr in in in a moment but anyway I think the asserts it's just it's just free uh for us if we run the East tests in the defet uh in in the in the defet as we have it um yeah exactly yeah exactly is East can produce asserts exactly that's that's okay cool um any testing blind spots you're worried about so yeah um basically I think we should update the checklist um and the main reason is that I think a lot of the tests if not all of them are currently imp

implemented in East um but mainly what I would like for us to have ready for next acde is just the is just quantify the progress so what percentage of the checklist that we have is currently implemented which I think could be close to 100% um and most importantly I would like to go over the eips uh and just make sure that every single um test case is actually in the in the checklist to begin with um I haven't got enough time yet to do this but I will try to do this in in this in this week before acds so we can have an updated checklist and

also missing test cases from the checklist also and just basically come up with a percentage of of readiness regarding spec tests um uh yeah yeah um cool so how do we feel about um I know uh Tim was pushing this do we think we're ready to add eof to test net 3 the next test net is that a question for me or for everyone for you or for everyone you could start and everyone could chime in I I can say okay yeah I I think I'll

I in my I I'll have a better opinion once I go through the the checklist and um um and in the in the rest of the aps uh more thly maybe in the next couple of days I I I will but please any you anyone else can can share me okay I think Peter was going to talk yeah so recently we put out a PR to add a a sort of testing Matrix which is clients uh versus different kinds of test that we now uh mainly to you know make to make sure that given clients are passing the test that we have before we proceed into

fing but this also applies to De I guess this should be like I guess the maybe the most basic checklist that we can take off right now so so in there we want to want to gather information whether given client passes a given version of test that we have right now do we need to be 100% before we go to a devnet I mean what's what's the advantages and disadvantages I don't know I mean for fing it makes at least you know the same sense is that we don't pick on we don't find issues that are covered

already but not used um I mean sorry uh so the fing doesn't pick up issues that are covered by tests but the client doesn't run those test right um I think it doesn't make sense for a client to join a devet if it doesn't pass the test or it's don't even run it so that this kind of but I'm not sure how much does it matter right because it's not like every client has to join at the same time but maybe we should make it clear that um that's kind of prerequisite for joining devet is to to pass the test

first I would say it's not only the East test was also the ethereum SL tests where some of the coverage is right now and I mean as the next tries dm1 tests which slowly remigrated over to East but um not there I think I think it's important for for to have at least most of the tests in East if there's any outstanding tests that are important that are only in etherum tests those we cannot run them with consume engine and I think it's important that we run with consume and so if you if you guys can think

about any test scenarios that are missing from East that need to be ported to the dream test I think I would yeah I I will I would like for us to make the push to Port them more yeah that's definitely our our goal but I think it's we are moving a bit slowly towards this so yeah definitely there's there's even big big there's even like eaps like full eaps that are not covered yet well definite three it's maybe best focus on AIP 7702 because there are additional changes needs to be implemented I would not like to get in

situation where UF is blocking other IPS but I don't know if other clients start started implementing it so another thing I've noticing is 7702 is very much going with the test and prod approach um because only Mario's WR reference tests for it none of the other 7702 implementers have gone for it again holding us to a different standard but you know we'll run with it um but I think that's a valid point we don't want to cause 7702 problems um but at some point we need to get into devnet we need to join it um so maybe we could present that you know we'll join when 7702 is stable um one question which clients are ready to that have a flag

and could join a test net today um re re in Basu okay um so I mean we get two that's that's really what we need to run a devet um okay how about Aragon 3 how close are they to

participating in tests like this uh think it's unclear if they will so think it's still still unclear if Aron 3 will be ready for pectra so I think they are discussing if they they they are able to switch to Aron 3 architecture for pectra so kind of replacing AR 2 or not but I think the main pectra work is done still in Aron 2 which is stable instead of these yeah is Aragon 2 part of these

devs right now uh I don't know okay but they a lot of issues on dev two they were not following if I recall correctly okay but at least they are trying right right okay so yeah I think I think that like it's probably uh unless this this will change I think the Arun to is the the client that we are expecting to to be in Dev Nets

okay um any other General commentary or questions about where we're at any other anything anyone else wants to say or any directions they think we're missing okay um August is here people will be on vacation but I'm fairly certain September will be kind of a crunch time um to make sure we can get into the dev Nets and be ready um for the discussions of when they start getting seriously talking about flipping test Nets is when they're going to have to keep or cut certain features so that's going to be kind of a crunch time for us so we'll need to have everything ready um I'm probably going to work on

the tracing spec and then I'm going to work on trying to write a uh Basu execution fuzzer so hopefully we'll have better signal there um but if anybody else has experience writing evm fuzzer could you know modify mariuses to work with eof that would be some some good stuff to do um any the comments cool I'd like to thank everyone for calling in just last last thing we should ask get another mine on ACD about UF progress because they're not like appearing here

and right it would be good to know where are yeah I know I know Maris is on vacation I'm a bit surprised nether mine didn't show up I was expecting them to show up but yeah we should pressure them for status and uh yeah yeah make it I mean they're going to complain that it's it's not being thoroughly tested but you know it is being tested you know we need to make sure that that if if it's not progressing it's clear why it's not progressing what it is so all right maybe I'll ping them we should ping them also on Discord cool great well if there's nothing else I'll give everyone three whole minutes back

thanks for calling in