All Core Devs - Consensus #64

Transcript

[Music] [Music] [Music] so

[Music] [Music] [Music] [Music]

[Music] [Music] okay here's the agenda um just quick updates uh particularly want to hear what's going on with altair progress um then we will discuss uh what's going on with altera with respect to spec planning that kind of stuff and then

if there's anything else to talk about then just procedurally

Danny Ryan: we are going to experiment with moving the merge specific call to right before this call rather than on offset weeks in an effort to help people with late time zones just kind of stack some calls um that will the first time we're doing that will be two weeks from now obviously it might be a long stretch um we'll give it a shot this calls usually short enough that i don't think we'll get too bad cool so let's go ahead and get started client updates

definitely let us know what's going on with altair how things are going with respect to getting things implemented both from the um state transition and also if you've had any chance to stand up to any test sets with all the features okay we can get go ahead and

get started with the lighthouse hey paul here um so regarding altair um we've been doing

Paul Hauner: successful tests with the networking side of things so like switching on and off um the correct gossip networks it's looking good um we're now finalizing the changes to the vc once we have those done and then them merged together then we'd have i think a full stack so to speak of altair so

looking forward to testing with that um and me personally i'm doing a pretty detailed review now um of our altair changes because we're going to be merging them into master soon and i want to make sure we don't break anything uh now and when we hit delta of course um that's about us our updates for our tear um regarding the merge we shut down our nocturne nodes that was good they ran without incident the whole time which is pretty cool so it speaks to all the um execution clients we ran as well we've done to them um published a blog post regarding our 1.4.0 release which has lots of features you can probably find you can find that in our blog probably get there by the github i assume um and we're still on weeks of activity um it's working for us but we haven't

got um backfilling done yet i'm interested in if other teams are backfilling to genesis i haven't been able to find any clear information about it um we're not going to release until we can backfill the genesis so we can keep those blocks going but they're interested if there's a a general um consensus that we're all going to do that that's it for me

uh protector

Adrian Sutton: is backfilling to genesis at the moment i'd like to be able to cut that weak subjectivity checkpoint kind of distance but obviously that depends on all the clients not needing to think from genesis and that not being a common thing and probably having some other systems in place to get the blocks so i

think for a little while at least it the back filter genesis makes a lot of sense and it's not a huge amount of disk space

Paul Hauner: yeah okay nice cool yeah that's that's pretty much along the lines of what we're thinking as well

yeah cool

Danny Ryan: i think the idea would be eventually to bound uh what the pdp layer is responsible for and have good methods for accessing historic stuff throughout that means but for now yeah it's safe to do so

great thank you uh nimbus hi

Mamy Ratsimbazafy: um so regarding altair uh we are currently with

the merge altair and the the changes we do to have a validator client uh that because we are revamping what we have already we are doing infrastructure changes at the moment um but we are expecting to pass the initial tests soon otherwise we released yesterday or v 1.3.0 so this includes in particular migration guides to and from nimbus and other clients something that was due for a while we have also

a new comment to test the validator performance so that people can ensure that their hardware will be sufficient to handle the load of the network we now support we have official binaries for macos and we also support april m1 so the new apple macbooks and mac mini we have activated for everyone pruning of the slashing protection database and optimized queries this led to a very significant improvement on uh disk io and cpu usage of all nodes and this will allow

us to have more validators on the same hardware in particular a raspberry pi and we have improved validation of attestations received from party source rest and json rpc api so that we don't broadcast and valid attestations when they are received from those apis and we have also improved the attestation subnet transition timings to improve cpu and bandwidth usage and in terms of problem we have also a gossip subnet working logic that was inefficient

and coming soon in next release and already being tested we have yet another um large i o improvement uh regarding to how we cache state this will be in the next release in the next week or in two weeks and that's it for me

thank you moving on to teku so we're

Adrian Sutton: looking pretty good for altair um we've been able to put together the whole package so battle data duties all performing and sync committee

um the networking stuff's in place uh the peer management was added just recently uh to make sure we actually find fears on the subnets we'll need and maintain those connections um so yeah we've seen that work with uh small networks of multiple nodes and been able to finalize and perform all the duties perfectly which is really nice to see uh paul harris has been putting up a whole bunch of prs for the standard rest api uh kind of defining the new parts of that around sync committees and to do with the different block structure and state structure um so there's i think still a few of them open um mostly based off of jim mcdonald's

initial proposal and just putting it into the open api spec so that's worth reviewing for those people working on the standard api stuff tech is implementation of those is uh pretty much done the one that isn't complete is the state stuff we got upgraded to the alpha 5 release just today which has the final change to sync committees in state so that needs to go into the rest api and it should be able to land um i don't know if it's useful for people for us to stand up a node that's starting to just run a chain like i have in mind just running a a really small test net but keeping a

note up that's got a public endpoint so you can test syncing against it since we've got any implementation that's useful for people let me know and we can probably get that up pretty quickly i don't intend it to be

a actual proper full-scale test net or anything um i think that's about it from us in terms of stuff that's relevant for you great that's awesome glad to hear all the pieces are coming together um yeah

Danny Ryan: i mean if you do stand it up i'm sure people will do some basic interrupt on it in the next week or so

but we can communicate offline on that one okay prism hey guys terence here

so on the tear

Terence Tsao: front we align to alpha five we are passing all the spec tests regarding the state transitions so now we're working on networking vc and the actual forking logic so we do face some unique challenges with a tear due to the lack of generics for go so we're a bit behind but but i think we can catch up pretty quickly so on the merge front we shut down all the notes and the validator instances so and the good job with all the teams working on that we um released version 1.3.9 last week it has a number of improvements including

updated go library to 1.16.4 also the proposal is using this um fancier mess cover algorithm so basically to be to be more profitable and we're also working out with subjectivity thing we're almost done with the checkpoint thinking and we haven't thought much about the blog that feeling yet and our production optimized slasher is to be released next week so definitely keep an eye

on that yep that's it thank you great thanks terence and lodestar hey uh so on the alter front um

we've got i think

Cayman Nava: all the pieces together at least the alpha 3 release level uh we've got a some simulation tests that test a small ephemeral test net for a few ebox working on just a single node and then a multi-set up we're in the next few days gonna be working on upgrading to the alpha five release and we've also been put to get putting together a light client prototype uh currently uh we've got a website that kind of can demo this but it hooks to hooks to a mock beacon chain uh because we weren't quite ready on the

altair side but it communicates over just a custom rest api to receive sync updates and request proofs and we will hook it up to altair when we're ready

Danny Ryan: nice can you share that link in the chat yes i can i grab that i don't have it handy i'll should i get it yeah i don't have it handy either um cool okay thank you everyone um so it sounds like also progress is much progress and that things are getting pretty close

most people are aligning to that alpha 5. um we'll shift into talking about where we stand on all that and into planning a little bit so alpha 5 had a lot of progress and refinement we got like a bunch of review on our team and some others to enhance testing and kind of do some vinyl tweaks to the feature set we do are slated to do an alpha 6 tomorrow which is primarily again more testing i think that all features are stable other than the what will be the addition of this resource unavailable error code which

is being discussed right here and um i think we need to get a pr up for that in the next 24 hours or so for review and release tomorrow that won't be a spec freeze but it will be an intentional freeze like future completeness and not meddling with things unless we get some feedback from engineering teams in the next you know few weeks as we are working on test nets that something's wrong or something isn't going to work in production as we thought it would so i think that alpha six will be that um

intentional target and then once we stand up once we do some like client uh multi-client interop um and validate that we're all running the same stuff and it all works um then we'll do a freeze and pick workdates and stuff um let me see i'm trying to pull up what we were discussing last time so last last time we discussed first half half of june for short-lived test nets um end of june for forking current test nets into july early august for uh the actual main net upgrade um i think we're definitely on target for that first half of june short-lived test nuts um and a spec a final expect release

then um as for end of june or early july for forking the current test nets and as for a target um end of july early august i think we should still kind of see how things play out the next couple of weeks uh but those still being the targets um and other than that i think uh maddie has been working on the fuzzing is there any update on that

maddie uh yeah yeah

things have been going quite well uh i mean with the a live beacon chain i don't think it's uh a good thing to probably disclose some of the books that we uh have been finding but we've been yet we've been finding a couple of interesting bugs over the last uh few weeks um

so we ate with the relevant teams and uh they're working on fixes but um yeah it's been it's been going well so we reached out to all team client teams and uh two of them are ready for uh fuzzers to start targeting altair branches uh two aren't uh just yet just fyi um so but um yeah things are going quite well

in that right

Danny Ryan: great so still looks like we can get some altair fuzzing running in june which should give us some good coverage by what the intentional fork date is yeah okay so

other than that are there any things to discuss with respect to altair

Protolambda: so this is not directly related to alter but comes with the same update maybe there is this one pr open to improve the config format so that we can improve the experience of developers running local test nuts and configuring their own chain it separates the presets the compile time configuration from the runtime configuration

i'll link the pr and it mostly just look i'm looking for feedback so we can get it right

Danny Ryan: proto is this something you're intending to get into the release for tomorrow

Protolambda: yes so i was announced earlier in chat ideally if you get enough feedback we can put it in the release tomorrow

and

Danny Ryan: for engineering teams this they don't have to change anything immediately essentially we still have the same values but it does allow them to over time migrate to to support stronger compile time uh constants and also have like configuration in a separate place

to match the spec over time

Protolambda: right so the variables are pretty compatible there's no change there no change to the consensus it's really just focused at improving the experience of configuring a client so think of all the configuration variables that affect typings or that go into the compile plane of a client those get separated from the runtime configuration so say for example if you run a nimbus client on a custom test net and if you adopt this kind of configuration separation then we don't have to recompile the client for every different

destination

Danny Ryan: right but if we get this out tomorrow

does it put any required immediate work on clients

uh just trying to understand that

Protolambda: the remaining open question is the change to the api or not there's this one endpoint that returns the configuration what we could just do is keep it the same and return the union of compile time and runtime configuration and then in the future we can either create another endpoint for the runtime configuration specifically or just drop the compile time parts of the response

Danny Ryan: gotcha okay um it looks like we have gotten some feedback from a couple of client teams if you haven't taken a look please do i think this generally makes

things cleaner on your end over time and that's that's kind of the intention so we're just looking for some sandy check thumbs up on that from various teams okay great um there's also been some discussion around this uh resource on available return code that i did share that's 2414.

um i'm going to catch up on that all right after this call jump in and try to wrap up this conversation pretty soon so we'll get that on the next release

okay research updates

Danny Ryan: as we are not having a um merge call this week or next week uh it will be the following week um probably a good time to just give everyone a quick update on where we stand on rainism and the merge and things like that mikhail proto do you want to give us an update

where do you want to go first sure so

Protolambda: regarding reaganism we just closed

the testnet yesterday it was a success we had four consensus clients on the test net three execution clients but going forward for the merge what we really need is to rebase on the production guard and implement the proper api separated from the user layer as well as um we need this uh specification for the merge transition and this is something uh michael amico will continue with

yeah so um yeah

Mikhail Kalinin: the plan is to work on the transition

in the meantime and also there is a work on the state sync that is happening as well um by the team which is great and while the clients are focused on altair and london uh we will have a time to do this research and spec work and get back to the merge after altair and london and yep one another probably bigger merged test net with transition process with the state sync with the other

probably tiny uh stuff signing up changes that we will have on the um research side yep but the major things that are going to happen is the transition process figured out and tested in the local test nets um and yep and same process as well it's important one yeah also there is the discussion about the consensus api but this is more uh this is rather more technical than um researchy so it should also be finished by that time

Danny Ryan: question about that um in the initial

state sync work does it look like that api is sufficient or do you think there might be an additional endpoint that needs to be added for the communication on that around that

Mikhail Kalinin: um yeah it's like uh according to the current state um of the arts uh the set head message is to be enough and like yeah the current message set is going to be enough for uh state sync but we'll see when it's like finished

Danny Ryan: right okay great um any other questions comments uh discussion points

on merge progress before we move on

Mikhail Kalinin: i just want to say thanks everybody who participated in the ryanism and in the nocturne which is the last which was the last devnet that was great a lot of excitement and progress has been done

Danny Ryan: yeah very much agreed okay um other research updates today

Vitalik Buterin: um i have uh i added the alters back to

my annotated spec repo so if anyone wants to look at that any feedback always welcome

great thanks for telling

Mikhail Kalinin: um also one thing we might have missed is that charging work will be happening after the realism uh so people are working on the implementation and proto will probably work on the stack side as well um also withdrawals uh we've been planning to deploy the

withdrawal.net as well so it's still planned and what will just happen outside of brandon's frames

Vitalik Buterin: uh is there a good place to find these back for the withdrawal design you'll be attempting

Mikhail Kalinin: yeah i will just drop it into the chat

Danny Ryan: okay um any other research related

items today okay great um general spec discussion testing discussion open discussion anything else people want to chat about today before

it was

Hsiao-Wei Wang: um i have a proposal about aligning the art air fork epoch with the thin committee period

um posting the pdr here i like to hear the opinions from the client desk so it's about the aesthetic consideration but also ensure that we won't set or we won't start with a too short think period when at the beginning of the folk but the drawback is that it might be more difficult to coordinate a reasonable track time for for people but i don't know it's a global project so there's no

perfect time for everyone anyway so open this pr for discussion but it's not urgent we still have time to discuss the folk epoch but just write the issue here

now and so it's

Danny Ryan: the primary benefit here to not have a truncated sync committee period because the transition function can handle i think starting anywhere right

it does

Jacek Sieka: i think one other convenient

um constraint that i've been thinking about is that we store i think it's 8192 roots before we accumulate them in historical right so it might be convenient to place it on a 8192 boundary

oh

Danny Ryan: interesting so you have um the same type throughout the accumulated chunk

Jacek Sieka: yeah exactly i mean this is related to something i've

been thinking about that if we have the block roots in there um in a separate accumulator we can actually verify the entire history of of of the ethereum two zingy with with these accumulators which would mean that they would they would be useful to identify like a chunk of 8000 blocks uh if we ever want to discuss alternative archive solutions for example right so one can imagine that it's a convenient chunk of blocks to keep together and identify by their accumulated hash and then you could

actually verify it with any states you can actually verify the whole easter history with any places if it were from that way

Danny Ryan: so you're saying historical roots rather than combining block and state keep them separate

Jacek Sieka: yeah exactly so you would have historical block routes it would be the merkle route of 8 000 blocks instead of as it is now 8 000 blocks in space um i mean yeah with the states you can't actually verify it

because the state group changes for the slot as well so that kind of annoying so you actually have to apply the block if you want to run a full verification on everything that led up to today

Danny Ryan: right right let's see someone can't just give you a bunch of blocks and you can verify them you have to actually have the associated state routes given to you or calculate them

Jacek Sieka: yeah exactly like that would be super convenient for for large arcade nodes or for you know putting ethereum on bittorrent whatever right because you

would you would have this very natural identifier to go by and it would actually be part of the date i kind of wanted to get this info for outside but i haven't had time to make the vr it's a really simple change borderline trivial um um i could actually type it out today or tomorrow but

but then there's the phrase right but not like anything i mean i guess the next step is definitely typing out and putting up

your discussion

Adrian Sutton: i guess just coming back to charway's original question around sync committee alignment um we found with the validated client that there were some interesting corner cases when it didn't line up but not not particularly difficult to deal with um so teku now handles um the the sync committee start or the sorry the fork starting in the middle of the sync committee and it just seamlessly goes through it doesn't really care um the main thing is is just being aware in a few places

of when you're looking for a state that you can use to to calculate the sync committees from or get the sync committees from you need to make sure it's actually in the altair fork and so there's just one more kind of min max type condition in there um it would make sense not to be too close to the end of a sync committee period um like i think we should either aim to align it or at least be kind of towards the middle-ish kind of thing or you know rather than necessarily being right at the end uh but we know those first few slots are going to be fairly awkward anyway so it's almost it's almost worth kind of abandoning that first thing committee period give it you know six to ten epochs of

setting up networks and so on so the next committee starts off and does the right thing

Dankrad Feist: um sorry what difference does it make when you're just at the end of the committee

Adrian Sutton: it's just in terms of those when you first hit the altay fork that's the first time you know the sync committees and so you've got to go and find peers that are also on those subnets people take time to broadcast all the metadata

and

Dankrad Feist: worst case it just fails so yeah like what's the big deal you just lose a few hours

Danny Ryan: adrian the first two sync committees are actually the same

well that's true yeah

Danny Ryan: so in that case it doesn't really matter because you see stability on the next side of the um

committee threshold as well

yeah so in that case it probably doesn't matter at all when we do it um because you have the same like you basically got a double length sync committee

Danny Ryan: right so you get a chance to find each other

yeah and

Adrian Sutton: and the first you know few epochs or however long it winds up being is is always going to be a mess because you're not going to find peers so your signatures are just going to get dropped on the floor pretty much but once that network stabilizes then it should pick up from wherever it is the one advantage we had was that we our validated client knows about the sync committee periods and we took that into account when we were calculating duties so it was then very easy to just ask for

the last epoch because the standard api is epoch-based when you're requesting sync committees if other people take different approaches to scheduling their duties then maybe that has different impacts and engineering-wise it might help but uh otherwise i don't think there's any great pressing need to to align them from my view

but i'm not against it i mean

Dankrad Feist: i think just like having to align it will just make four coordination so much harder and because like you want all these properties of the fork it should be at a convenient time blah blah and then like you'll find that rapidly you only have like a few windows every few weeks or something where

you can do it

and i mean that's just not ideal

Adrian Sutton: was your only concern just that it it was kind of untidy or were there was there something else in your mind

Hsiao-Wei Wang: not particularly so just wondering if people think this is a good benefit for um if we want to remove the the half period from the beginning but if it's not a

big problem for the kind of deaths then i don't have strong opinion on it also it was found in the previous pr feedback so just wondering to uh do you want to get feedback from you guys

Dankrad Feist: i mean clients could also just ignore the half period at the beginning and just say like whatever like i won't do anything like i don't see that as a problem

yeah

Danny Ryan: but i mean if if it wasn't too bad it was pretty easy to do the exceptional logic and the fact that it's the same committee on both of those first two periods um those would lead me to believe that this

is probably just fine to leave houses and be a little bit more flexible on the

floor

Hsiao-Wei Wang: that's good good to know so yeah i think for the sync committee period i think it's fine to just uh not totally ignore it but we don't have to make it to be guaranteed like we don't have to align it

in this fake but uh do we still want to consider the accumulator

Danny Ryan: um that one's even tougher uh because that's like 32 days or something which would make it very difficult to align things i don't know um maybe we should talk about that one offline unless people feel strongly about it one way or the other

Hsiao-Wei Wang: sounds good to me

Danny Ryan: cool

other items for discussion today okay i mean i would suggest in the coming week or so to uh try some like basic interrupt stuff um i think that's probably the next natural progression in this thing and then we can stand up stuff that's a little bit longer lasting so we will talk on the internet and talk to you all in two weeks thank you

thank you thank you thank you [Music]

[Music] [Music] [Music] [Music]

you

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.