All Core Devs - Consensus #73

Transcript

[Music] [Applause] [Music] so

so [Music] [Music] [Music] all right

stream

Danny Ryan: stream should be transitioned here's the agenda major goal is to make decisions on altair as much as possible thank you

members of rig

Danny Ryan: members of the research team and members of client teams working through data analysis and bug fixing around sync committee performance proto has quite the graph on that issue if you want to look and the fact that all of the software that all of the nodes that we run look blue at the end of that graph is is

a good a good sign and as far as i know our sync committee troubles have been fixed does anybody have any other comments on the sync committee investigation any outstanding issues any important things they learned fixing those issues okay so

i mean

Jacek Sieka: i can just briefly mention the problem that we had yes it was kind of funny so we have this

anti-dos protection and lip b2p which adds some extra filtering because when people subscribe to gossip technically you could send lots and lots and lots of subscription and fill up our subscription table and like make numbers use lots of memory so we had a special filter there and we kind of forgot to add the sync topics in there so it was kind of sad as far as bugs go

Danny Ryan: oh i guess one question is are we do we have sufficient monitoring in place now i know we have sufficient debug tools um but will we be aware of issues because i i felt like this issue

kind of sat for a while without anybody maybe being aware i'm not sure if that's the truth

Jacek Sieka: i mean we were kind of aware but we're looking in the wrong places so we've got we're looking into that filter as well to make it louder

Danny Ryan: cool any other comments on the investigation issues found where we stand with respect to this feature we're generally good i i know that nimbus had uh i think there were a number of patches not just nimbus um we

don't have to go over all of them but just anything else we want to discuss here okay so last call two weeks ago we had a number of things we wanted to get done additional testing on pyrmont which perry carried out some i believe we had a release of additional test factors in that and we wanted to fix the altair uh the sync committee performance which came down to the wire we did and if we did that we wanted to talk about a fork date today a fork date

would imply client releases um mainnet ready client releases no later than two weeks prior maybe 16 days prior so that we can do a a blog post 14 days prior at the minimum to get people ready for an upgrade tentatively we talked about doing releases at the end of september that's the end of next week maybe doing a blog post a few days later and doing an upgrade mid october maybe october 18 20th something in that range

um where do we stand on that are we feeling do we have the same amount of confidence we're ready to do this do we have new information and need to make a different decision

Ben Edgington: uh speaking of teku ready to go but would prefer to see three to four weeks um lead time just because uh there are um the big operators will take longer to upgrade i expect and there'll be more due diligence around it i want to make sure everyone's got time

Danny Ryan: so three to four weeks from releases plus blog post to mainnet date right yeah okay

okay um so releases but y'all are still on the we can release kind of pre-october and then right right on the cusp of september october and then a three to four week lead time from that

Ben Edgington: yeah i believe that's doable yep for us yep

Danny Ryan: got it um other anyone against such a release timeline especially on that you know release by end of september okay

so let's uh let's pick a date my calendar i think kicks me out every four weeks at the exam same exact time and i have to relog in i think it's literally during this call because it's happened another time i'm trying to log back into my calendar i apologize

Micah Zoltu: set a calendar reminder to log into your calendar tomorrow

Danny Ryan: this is ridiculous i was literally logged in i think eight minutes ago okay i'm back in we're back in i can see it okay so we have uh the the end of next week that thursday friday is september 30th september and

october 1st so we could target our releases for the end of next week and a blog post that monday october 4th three weeks out from that is the 25th plus a few days to get into the middle of the week is the 27th 28th do we do we prefer do we have preferences on wednesday or thursday for a fork day someone's gotta have a preference

Paul Hauner: middle of the week sounds good today

it's further from the weekend

Danny Ryan: wednesday great and if we do thursday it'll be almost friday for you what was that terence

Terence Tsao: no i i i i was going to say a wednesday

sunscreen

Danny Ryan: okay wednesday the 27th we can pick a precise fork uh epoch using adrian's sweet tool right after the call and make a pr to configurations does that does that sound good is the target what was that

i missed that micah i'm sorry okay um okay so say it out or get out loud again end of next week client releases mainnet releases a blog post by the ef and anyone else that wants to join in that um on october 4th to discuss dates upgrades and client releases that then three and a half weeks from that point is october 27th which over the next day we will select an epoch on that date

anyone against what i just said okay let's do it um any other altair discussions or comments before we move on great let's go through some client updates we can start with teku

Ben Edgington: right design um let's start with introductions so we've

had a couple of new joiners uh reggio's been around for a few weeks and is in australia and not here uh enrico joined last week and is in europe and is here so welcome enrico on the client side uh we fixed an issue that came up uh on prior to network we first saw it there um in there was a rare edge case where teku would fail to produce blocks um it's some kind of weird race condition where we would see an attestation first in a block and then later receive it via gossip and then there was some other sort of weird conditions and when it all coincided um the we've uh failed to produce a block

um later uh it was rare on prata and we've not seen it on may net or had it reported but nonetheless it's fixed in 21.9 which is the most recent release there and other minor things um we've upgraded to blast 0.3.5 which came out last week we now have support for building on jdk 17 which is a newly released long-term support version of java and we're working on merge support in a dedicated branch and various other minor bug fixes and bits and pieces as usual uh that's all i've got that sounds like a painful bug to debug

adrian is uh is astonishingly good at debugging

Danny Ryan: excellent um prism

Terence Tsao: yeah hey guys so on the a tear front we're just chugging along we have merged everything into the canonical develop branch so that's done so right now just a few minor budget fixes with the rpc endpoint and then a few optimizations here and there so so far is looking pretty good on our end and then we are also gearing for the v2 reduce which which will be end of next week and they're reduced from 10 slashers we also reorganize package structures to be to be more draw-like

also align all the uh matrix namings all the system standardization and all the good stuff and like tech who were also working on the merge spec support on a dedicated branch and yep that's it from us

Danny Ryan: okay thank you terence nimbus

Mamy Ratsimbazafy: hi uh so we've been working on uh passing all the new tests uh also the one from blst as well or last week was focused on making sure that pratya worked so we had issues regarding sync committee messages and

also a low number of peers that we are debugging or have debugged and uh we want to do a release next week well so the the timing is good uh to fix everything and make sure that or main release is rock solid for pratter and altair

Danny Ryan: great thank you lodestar is anyone from let's start here um loadstar progress continues uh they

have a mainnet validator that i think is is doing quite well these days um and continued r d on light clients and getting altair refined next up lighthouse

Paul Hauner: hello paul here um so this week we merged two big pr's into our unstable branch uh so they were weak subjectivity sync or checkpoint sync so we can now um with a trusted uh api endpoint we can now sync from scratch to head in less than a minute which is pretty cool we've been playing with that and having fun uh we also emerged batch uh bls verification and attestation signatures

so that sees about a 40 to fifty percent drop in cpu load average on our prada nodes so that improvement uh is significant on knows it is subscribed to all subnets um still noticeable on other nodes but much less so so we're likely pushing a release candidate of version 1.6.0 next week and that'll include those fixes michael did some analysis on client distribution across validators by fingerprinting attestation packing characteristics got some attention on twitter was pretty cool we're working on our implementation of the merge getting ready for more test nets very soon um and adrian manning has been working on reducing the p2p bandwidth he seems to have over 50 reduction at this point but we're still monitoring to see if it

comes with validated performance costs that's it from our end

Danny Ryan: on the lib p2p bandwidth do you have a tl dr on that

Paul Hauner: i haven't been paying attention i think he's trying to reduce gossip uh duplicates by reducing his mesh size or something like that um and not super clear on it

got it thank you paul and grandin this is solus from the nina team so

Saulius Grigaitis: so we are still focusing on the uh multiple runtimes refactoring and regarding the alter er we can sink the chain now and

we still need some work on the validator side and i think maybe in a couple of next weeks we will be back on the broader

Danny Ryan: got it thank you okay uh moving on merge discussion um we are working very diligently mcguile myself and many others on refining initial interop specs for the merge we expect a release tomorrow on the consensus side which would be kind of a stable target for

initial interop in the start of october um additionally you know on the on the execution layer side we'd have that eip stabilized and also there is a minimal version of the execution api of the engine api um here that's being put through the ringer and we expect to stabilize again in about 24 hours for release so that we can all have a common target moving into october um that's the main merge update on my end uh anything else mikhail or others

Mikhail Kalinin: ah thanks danny for the update i don't have anything to add here i was just going to say something like that

like you already did

miguel

Danny Ryan: are you going to do put all the links kind of in a single place as something like a meta spec for initial

Mikhail Kalinin: oh yeah right right yeah yeah um i've started to um to work on this document it shouldn't be that long i mean just uh spec versions and fill in the gaps that we have in this pack um like what what to do with the random field on the execution client side so because we don't have any eip for for that yet but it's already specified in the consensus specification so cool that should be finished tomorrow within the

spec release i guess

Danny Ryan: great and if if you all haven't taken a look i mean primarily what we're dealing with is upgraded types and this communication protocol from the the consensus side so order of magnitude simpler than altair i would suppose especially now that clients have kind of a standard fork mechanic path in the code base so take a look any questions about the merge or comments or discussion points

i'm

Paul Hauner: keen to get my hands on an execution node that has the api implemented um

yeah if anyone has that kingdom key to play with it

i'm

Danny Ryan: not sure who is farthest along um hopefully that would be something relatively soon pro and others were thinking about actually mocking an execution engine or mocking the consensus side for testing of the api proto do you want to talk about that a bit

Protolambda: right so what we learned from rainism is that once you have many clients talking to each other you have this quadratic integration issue which can like really like you it can delay things and so instead i would like teams to focus on sharing more tooling and trying

to share testing and this one of these things that we can share is a mock version of the beacon node a mock version of the engine mod that conform to the specification are lighter run and you can test against and once we can have all consensus clients work against the mock exclusion engine and our exclusion engines mock work against the mock consensus clients then we have a much better chance of interrupt

Paul Hauner: yeah i totally agree i had a lot of trouble last time trying to get matches because the serialization formats between um the consensus clients and and

execution clients are quite different just in the way that we you know serialize byte string hex strings and stuff like that so it'd be super useful i think i published some test sections last time i'll do the same once i can get it up but yeah a mock thing would be really handy i was going to produce one on our end because we're going to need it for unit testing as well in ci so we don't have to spin up a whole other execution chain

Jacek Sieka: i mean on that note one thing that i've been taking mental notes for and and some real notes as well is that uh given that the execution api is kind of this private ish api between the big like the consensus client and the

execution client they shouldn't really be exposed to user applications like normal user applications and that means that it kind of could live in a separate space in the execution client now for many many reasons we chose rest to talk between um you know vc and bn and and also for encoding purposes it would make a lot of sense if we continue to rest to use rest between consensus and execution but like before making that pr

is there anybody that's going to like loudly scream and inject right now or is there something worth considering i think it would be a big simplification for the future if we want to kind of focus on just one kind of encoding and one kind of um api style tommy

Danny Ryan: the primary pushback is the execution layer already has a json rpc interface it's it's suggested to be exposed on a different port for some sort of separation but at the end of the day it's a compromise in one way or the other in terms of formats required because the the user api on the json rpc

is not going to go anywhere um i this has been debated a bunch i'm i just wanted to provide that for context i i'm not going to throw my hand in the ring too much

Paul Hauner: my understanding is that we used an rpc because this type of communication is not suitable to rest we need because we need the like we can't really load balance um the execution node because we need to stay in sync so eventually we're going to have a protocol where the execution note is going to be like oh no i actually don't know that block that you were that you refer to and then we have to push blocks to and we have to like create this very much rpc style sync um method between the two that was my understanding is why

rpc was chosen

Jacek Sieka: i think they're fairly similar in that aspect i mean if you read the two specs there there's very little actual difference between them except you know everybody has their pet favorite um the reason why the reason i'm mentioning it even is like one of the heaviest arguments in favor for rest when we were choosing it back then was was actually like experienced from ethereum one um if you look if you go back to that discussion when we were choosing rest it's actually

like the theorem like the nowadays execution client developers that were one of the fiercest advocates of it

Danny Ryan: right but one of the primary reasons was the uh because it's a user api that might have need load balancing and all sorts of other um nice things that are going to fall out of a restful api whereas this doesn't have the same requirement i think is what paul's arguing because it's more of a one-to-one relationship but um again i i'm not gonna go too deep in here i i i know it's kind of gone back and forth a couple of times and ended up setting on json rpc

for relatively for simplicity because that is already just embedded in execution layer clients

but

Mikhail Kalinin: um one more argument here is that rest is tightly coupled with http protocol while the json rpc could be implemented on top of any communication protocol like tcp websocket yeah i guess the rest will be done in websocket as well um but i don't think that rest suits us well here because rest is good in accessing and updating some resources while this communication protocol is more like

updating the states uh and syncing the states between two clients so it's not always not each request and response corresponds to some uh like resource some logical resource

that

Danny Ryan: from a practical standpoint of just getting this api ready to be used tomorrow for uh people to build and then uh initially do some interop at the beginning of october i think there's probably zero chance it's going to change but if there is a compelling argument and you want to continue to have the conversation and try to change it after that's probably the path i'll sleep on it

cool anything else merge related yeah with the release tomorrow there will be um at least kind of a minimal set of consensus vectors available on the consensus side so keep your eyes peeled any research updates other than

emerge

um yeah on our website i wanted to

mention that uh our paper on the ethereum ii network crawler has been accepted uh for publication in the ieee international conference on blockchain computing and applications that should be held in estonia in november i will add the link in the in the chat and also regarding the crawler uh just a couple of days ago there was another um client distribution analysis that came out done by michael i guess many of you already saw it it's a completely different method that is derived from block proposal data and the distribution that came out it was astonishingly similar to the distribution that we showed in that we demonstrated with our crawler

from several months ago so i think this kind of validates a little bit the data that we have got and shown over the last few months i'm gonna add the link to the twitter so you can take a look at it and yeah on the the other paper that was accepted uh concerning the entering plans um resource utilization is going to be presented next week uh at the conference on blockchain research and applications for innovative networks and services i'm adding the link to the program in case somebody is interested to take a look i think the registration is free um yeah so that's that's research updates on my site

Danny Ryan: thank you

Mamy Ratsimbazafy: regarding crawlers uh there is uh one that underestimate nimbus or always connect to nimbus node i don't remember the team that uh yes right so there is one thing is that there is a difference uh like some clients allow crawlers to connect and then eject them while numbers don't allow them to connect at all for example and there are different behaviors between clients that makes uh statistics a bit tricky sometimes

Danny Ryan: right and i think cerium is actually maybe having a very large nimbus pure

store and dumping it um and so they're seeing like actually what you see from that is like biases and how nimbus connects to and doesn't connect to clients over time i believe that that was my understanding of it when i spoke with them yeah and i'm actually surprised how close the michael sproul validator metrics were compared to the crawler metrics we've seen i expected some more asymmetries um between size of validator allocation and and the nodes in the network paul

Paul Hauner: i was just gonna say that i was talking to the esteriam person as well and they

were using it uh they were dumping in nimbus peer story i think they had like 40 percent of the network was nimbus or something like this um yeah i think i mean trying to crawl is very hard um and people need to really consider think very hard about it when they do it not just dump peer stalls and cat peers

Danny Ryan: agreed okay anything else um any

open discussion closing remarks anything else people want to discuss today um i'm going to suggest that we do not have our meeting two weeks from now um because the number of us will be in person working on merge interrupt um and altair will be kind of progressing along i would say we only would would do a call or some sort of formal gathering in the event that we're having uh some sort of issue or hiccup with the altair progression at that point and then we would regroup again on the

21st of october which would still be before the altair upgrade for any kind of final or emergency discussions then cool

Jacek Sieka: uh i would have one question actually to client teams i mean we're still seeing a little bit of missed attestations and so on i was curious uh what the latest investigations point to i mean orphan blocks were one thing so when those things were fixed but if there's any news on that front maybe releases that address um potential issues and cure incoming attestation queueing things like this

Terence Tsao: yeah so on the prism front we have a bunch of optimizations that improve that and it was released in the previous release but those were actually part of the feature flight set so basically in order to use this optimization feature user needs to enable the flag just just so user knows what they're doing so now with enough testing for our v2 release those flags will be flagged uh those flags will be flagged or into into the default state so most of the other transactions will be enabled so we should definitely see some um improvement there from our end

Danny Ryan: yeah so specifically what we saw and some uh analysis left by barnabay was that there are these like zero uh zeroth epoch zero slot blocks um that are late um and and kind of hap increasingly so that cause issues with voting on that epoch boundary cusp and that from what we can tell is is primarily the drop that we've seen over the past handful of months and using graffiti analysis it was did generally happen when uh prison validators were voting were proposing at that epoch boundary

and what terence was alluding to was they have quite a few optimizations um for that epoch boundary one of which is just not waiting just in time to do it and and kind of when you're at the prior to that boundary and you know kind of what to build on you can optimistically just do that epoch transition get the shuffling in place and stuff

i

Paul Hauner: think um i know micah was looking into it as well i think he found some cases where clients weren't packing um attestations to blocks as well as they could i think he's reached out to those teams and they're aware so and doing stuff about it

Danny Ryan: yeah the nice thing is we have a required upgrade in late october where any of these optimizations that have been kind of filtering out will all be enabled at that point so that would be you know our our data point as to whether what we've been putting in place on packing and putting the place on these zero slot optimizations uh epoc transition optimizations are actually going to fix what we think they're going to fix anything else on that

Jacek Sieka: all right cool

thanks

Danny Ryan: other discussion points okay cool thank you um we will regroup in the chats to figure out a fork epoch and get that up in the configs uh good work everyone and uh releases at the end of next week talk to y'all soon thanks all bye

thank you thank you [Music] so [Music] so

[Music] [Music] [Music] [Applause]

[Music] so

[Music] [Music] [Music] [Music]

[Music] do [Music] [Music] [Music] [Music]

[Music] [Music] [Music] [Music]

[Applause] [Music] so

so [Music] [Music] [Music] [Music]

[Music] you

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.