All Core Devs - Consensus #72

Transcript

[Music] [Music] [Music] [Applause]

[Music] great

Danny Ryan: welcome to the call here is the agenda i think probably the most pressing things are to discuss altair and altar timelines and what we need to do to get from here to there um to that end the first thing we will talk about is altair uh prada forked um i think generally successful and there were a couple of bumps that were ironed out relatively quickly um and i think one of the main bumps was self-healing after like an epoch or so uh which is good um i

if you take a look at the pratter explorer or the pyramid explorer we're still seeing lower than expected sync aggregate participation um i think that we certainly need to do a bit of digging and analysis on if there's a pattern to who is being missed this is a single client issue or if there's it's more in the aggregation protocol itself um so we'll try to look into that over the next couple days and see if we find let you know um any comments or discussions on prada okay

um in light of the standing issue with stink aggregates um that we still need to uncover um and after some discussions with the prison team who is simultaneously trying to release uh altair and their big v2 um i think there's been on there and a request for at least a couple of weeks delay so rather than doing releases next week and fork at the end of september do releases at the end of september and fork mid-october um i think that generally although we need ship we need to get it

right so i lean towards doing that um but also simultaneously during this month shifting engineering resources to ensure that we have kind of the next wave of merge implementations ready to work with the execution side because they are heavily in progress and i want to make sure that altair does not begin to be a blocker on that or that the the consensus side doesn't end up being a blocker on that um also i think we are continuing to enhance test coverage on altair and

our aiming to release today but then maybe even another one at the end of next week um as we continually kind of go back and forth with the clients on on different edge cases and books that we're identifying general planning other than extended time and identifying the issue on the sync committees on the test nets that we're observing what else needs to be accomplished in this next month some some what are the musts on your end

Terence Tsao: i will say just more enhanced testing on test name

by enhanced testing i mean slashing deposit withdraw all these uh all the manual operations and will be fun to trigger some like chaos monkey tests with with timing such as like sending the blocks really late sending the attestations really late and just see what happens from there

Danny Ryan: right so pyramid especially for the chaos monkey stuff uh might be the better target um i think one of the easiest things we could do is force times to be off uh so force subsets of nodes to have their

clocks skewed um i think that would induce quite a bit of what you're uh alluding to um which i think probably we could do without too much issue we we do still have this remaining uh let's see i think it's in these two clients repo youtube networks repo uh there still are a few things to get done on pyrmont under scenario testing um if you do have oh if you do have updated releases please make sure you're getting them out on

prader and pyrmont um i know that a number of us have discussed the shifted timeline outside of this call i do want to leave some room if people want to chat about it a bit more here obviously no pressure either way on that

Adrian Sutton: so is that saying we'd then try and set a fork epoch next meeting or the meeting after

Danny Ryan: let's see

i mean we could i think it might be uh my calendar is asking me to sign in i'm sorry give me one second i'm trying to see apologies i i mean if we are going to do releases in the last week of september then we must pick a fork two weeks from now

Adrian Sutton: yeah i mean i'd rather sooner rather

than later at least for picking the fork even if we pick an epoch a decent way in the future

Danny Ryan: yeah i mean we could do that right now i'm especially with the sink aggregate issue not resolved i'm hesitant to put it on the calendar

yeah i think that's reasonable okay

Danny Ryan: okay um after this call i'm going to

circle back with my team and client teams and see who wants to do some of that investigation on the sync aggregate bits um and what we can do to make sure we get through this list on pyrmont and any other sort of uh wacky testing cases uh like messing with our clocks and things okay anything else on altair with respect to planning

onward thank you

let's run through some client updates we can start with load start um i'm having a lot of trouble hearing you lying are you can you get closer to the mic can anyone else hear lion i'm hearing just the faintest hammer very quiet yeah hey now it's better yes thank you yes sorry so yeah

dapplion: we started to implement the merge

not a lot of progress but working on it we have also reduced the memory cost of our states by half so a lot of less gc and we have shared uh documents with our proposal for implementing our client server with teams we got some feedbacks and we'll appreciate anymore uh also reach out if you are interested to implementing that that's all thank you

Danny Ryan: excellent thank you nimbus

uh hi for out for the prattle rounds we see another beta for outer beta

but we are now working on preparing a stable release by testing the same code on mainnet the other other notable feature that will be included this new release is the multi-core vos verification support that we've been working on which will be initially an opt-in feature we've also prepared system packages for linux and installers for windows and mac os uh that will be also shipped in design release and they these installers set up a big node service on the user machine automatically after installation on the development front we've added acc payloads to the rest api and we started work on a light plant as well

the main focus i guess for us for the rest of the month will be preparing for the march and the work here will be both on the nimbus seat 2 side and the nimbus 81 as well because we want to kind of have the capability to operate a network capital merge with both clans we've also hired a queue muhammad who should be known to some of you who'll be working to help us create a mobile ui for monitoring and managing your beacon node

very nice uh grand dean

yes this sort of engineer team so so worked on some optimizations and the default focus is still on this multiple runtimes to to accommodate the alter artwork and it looks like we are getting closer to having the working version of that that's all thanks

Danny Ryan: got it and lighthouse

Paul Hauner: um so we're starting to prepare for our next major release which is going to include weak subjectivity sync so we've done a bunch of successful syncs this week which is cool um it's also going to have support for the web 3 cyber remote

cyber we've also been spending a lot of time on analysis of mainnet we've been working to try and understand why lighthouse seems to not receive some num consistent number of attestations we also identified some issues with other clients regarding station packing inefficiencies and they've communicated those we've been running analysis on late block proposals we identified in an outstanding staking pool and contacted them we're not exactly sure which client they used i haven't got much detail um we've made further optimizations in the lighthouse to reduce the impact of late blocks like we added some more metrics and we've also got a special server-side endpoint to try and track um the peers and graffitis and frozen indexes that late blocks are coming from

and we're also starting work on implementing the latest merge spec anticipating a new round of test nets next month that's it for us

great um on the uh at the stations that you're not receiving those are actually you've figured out they're they're not hitting your mental at all so if we

Paul Hauner: we did this thing where we're like shadow proposing blocks so michael's been working on this but so he's um when we see a block we also propose a block and compare the attestations um and we can see that there's some left in hours and then if we go and take those additional ones and feed them into our

mempool then we do include them so yeah we're still we're we've kind of cleared it from the consensus layer i guess and now we're digging deeper into gossip sub trying to figure out what's going on

very interesting okay thanks paul and taku yeah hi so we've

Adrian Sutton: had a few key things going on so we had an issue with the altar upgrade on prada where we were shoving some invalid data stations into blocks and then correctly rejecting our own blocks which was good um so that's been fixed up it was uh an interesting interaction with the gossip validation and signatures um where we had a

state before the fork activated so yeah we're picking up the data station should have been signed with a different fork id so we fixed that and we actually no longer use uh in the spec there's the get domain where you pass in a state and it'll take the fourth from the state we've removed that function from tekku entirely so we're always very deliberate about fork so that we don't hit these kinds of problems again um we've put in a bunch of fixes for at a stations that were for redundant attestations that we were including in blocks so that should help with some of the issues on main net when we get that out in terms of making more space for other infestations and

reducing inclusion distances um and paul harris our paul has been doing a bunch of working in digging into where we're seeing duplicate outer stations um and the value of the caches at the application level uh so what we've found is that essentially anything that is going to be detected as a duplicate is done at the gossip level through the lib p2p scene cache and nothing useful ever gets caught at our application level um primarily because the application level just sees so many different attestations that it overflows our current cache straight away and we'd have to make it really dramatically bigger and led p2p is already

holding onto the scene messages for 32 slots which is as long as they should be valid so we're going to do some testing on prada and piermont um with not having that cache uh at the application level and just depending on the scene scene messages from pdfp uh i think lighthouse has mentioned before that you've seen some some adaptations that are coming very late and we've certainly seen that as well it's a small number but they're well after they should have been invalid um and it's kind of odd little duplicates that are filtering through there um

beyond that a bunch more um optimization work that's been going on um particularly in terms of the number of calls between the validated client and the beacon node um stuff like that uh just kind of carrying on and improving the world which is good so uh we'll probably get that into releases probably relatively soon if we're not sitting in our tier 4 epoch but we'll sort that out keep an eye out

Danny Ryan: got it thank you

adrian and we we have been seeing sorry i was just gonna say

Paul Hauner: we have been seeing lots of other testations around i think um lodestar has also noticed old blocks floating around we haven't looked

into it but i do remember having to make some changes to be more defensive against those i'm not sure if michael communicated either but we've also identified an actor on the network that was sending duplicate um aggregate attestations as well um which are probably making people's cases struggle as well

um so yeah some points define duplicate aggregate attestation uh so basically producing to

Paul Hauner: the same so two aggregates for the same epoch from the same aggregate aim aggregator index with different values different

routes

Danny Ryan: gotcha and consistently like many of them

Paul Hauner: yeah i believe it's consistent i didn't look into it but i believe it's it's very consistent we've reached out to who we think it is

are

Adrian Sutton: they at risk of getting slashed you'd do that if you had the same keys in two places it's one of the things where your attestation would appear the same but you're aggregated very likely you're different yeah you get away with it until you propose

a block that was my suspicion i don't believe so okay interesting early warning sign though

Danny Ryan: okay and prism

Terence Tsao: yep uh hey guys terence here so last two

weeks we have been mostly are merging a tear code from developer branch to the master branch making really good progress on that we went from ten thousand lines to two thousand lines and then we are also going through intense code review of the consensus code and the code that uses cash so those are mostly what those those are mostly where the big issues are and then we're preparing for our vt release we're cleaning up all the features and getting ready for those to be merged to the master and then i just want to give a shout out to the beta 3 spat test for the people that work on it we found the consensus

spot regarding the validator life cycle when processing rewards and the bot has been fixed and then and it is in the latest release and that's on all the test nets we're also reviewing all the test coverages regarding validator life cycles and then regarding magnet we have been reviewing our cash implementations when we first implement those cash we were prepping the validators to be around 300 000 ish and now the valve data sizes has grown tremendously so we're reviewing all the sizes again we're doing all the uses and we're doing all the usage again

and then last but not least we have uh a few new hires uh james which is on this call and james is working on the front-end ul with us and then zahu which uh some of you guys have met already and who is working more on the protocol and the and the research from with us and that's it thank you

and terence you all

Danny Ryan: also had kind of an over packing on attestation side too and and that went out until release last week

yes that did so if we look at mainnet we should between tekku and prism likely see many much fewer 128 blocks i guess over the next month and a half certainly

after pratter because people everyone have upgraded yeah that's right

Danny Ryan: okay cool

Adrian Sutton: actually i do have a little tool that's been monitoring the redundant data stations little node app um it could do with some cleaning up but it's it happily logs each time a new block comes in with redundant data station does a pretty good job at detecting them it's kind of scary how often they come out but hopefully we'll yeah we can use that to see it go down a bit more accurately than just at

a stage account as well

Danny Ryan: yeah i imagine we're over two-thirds on redundant at the station blocks right now

before the upgrades kind of ripple

out

Adrian Sutton: uh yeah most blocks seem to have some redundancy um it's usually kind of a couple of overlapping attestations in there um uh could be worse but every now and then you kind of just see a whole bunch scroll by i haven't looked at numbers to do deeply because it doesn't log to a database yet just kind of prints them still useful

Danny Ryan: okay moving on uh merge discussion i just want to let you know that later today i will i've been working on a document to help coordinate um

our initial targets for devnets between consensus and execution for merge devnets in october and i will share that with you all today so that we can begin to be on the same page and kind of have a target for that um in other news there was the continuation of the engine api discussion this morning led by mikhail you can see his latest doc uh has all of the latest and greatest although i'm sure a bit came out of that conversation and some changes that might ripple uh through there there's like a core set a minimal set of this that will be targeted for the initial wave of devnets

and we'll make that clear other merge related discussions for today

Mikhail Kalinin: um i'd like to share a couple of things um first the imagery from the xrx started to work on test vectors for the merge part of the beacon spec this work has been initiated by proto during reaganism so dimitri's goal is to add more test factors for those methods that are not yet covered by tests and also some execution there's some code branches that what it covered uh in already um

in the mess that already has unit tests so yep the target date i would say next week for it's been ready

i mean the pr

Danny Ryan: unit tests of some of those smaller functions or these like consensus test factors that would be released to clients every bit of both

yeah yeah the these are

Mikhail Kalinin: um yeah the goal is to have the consensus test directories uh it could be released uh yeah but it will be with the stopped uh execution engine it could be so yeah it could be just adopted by

consensus clients without any other external dependencies now the other one is there is pr i'll drop it here which proposes to use um the previous and the random mix from the previous slot instead of the random mix of the current slot 4 as a random value that is passed to the execution client and eventually included into the execution payload there is the rational in the pr and the discussion related to this um so

Danny Ryan: right

another point um so i i at least put something of an argument uh against it i think i've softened up and now in favor of the pr um and another point in favor the pr is an earlier design goal which was to be able to rip the execution layer out into an independent shard if if we wanted to um and the the lagging of the randomness that's embedded in there would also help enable

that goal oh

Mikhail Kalinin: yeah that's interesting yeah like the initial goal was to uh one of the goals is to like

make it future can make this design future compatible with the proposer and builder separation uh without making it too complicated to um for builders and proposers to to be uh on the same set of uh data that are required for building a payload uh like if we use the current random x then the proposer will have to communicate these uh randall mates uh upon receiving the parent block uh through some channel to all block builders and actually it means that it anyway reveals the random field in advance

so this kind of stuff is like one of the reasons for this

vr

Danny Ryan: i'll catch up on the uh the thread i just noticed there's a bunch of new comments and um we can try to get that through soon thank you pick iel any other merge related discussions for today great

any general research updates got it

um any other discussions for today spec related open discussion any closing remarks

Mikhail Kalinin: um i have like a small question uh how do we want to proceed with the engine api stuff yeah where do we want to make another call

Danny Ryan: i'm a bit resistant to doing another hour call where we walk through it i um i think instead it might make sense for this to go into um maybe the execution apis repo or an independent

repo where we can probably do a pr per set of methods and take some discussion there and just start hammering it out uh yeah allow more people to engage with it too

Mikhail Kalinin: right and it also makes sense for to crystallize the minimal set of methods to be implemented for the merchant europe stuff

Danny Ryan: yeah i mean ideally a week from now you know we have something in a repo that we can point to and even if it still might be iterated on uh people can begin to to work on some implementations yeah cool let's do that do we want

i know a couple months ago we thought the execution apis under a different name space would be the appropriate place um is that where you're thinking

oh

Mikhail Kalinin: yeah probably now so this this this dog yeah i was like looking at the rbc stuff yeah the execution api is there and it's just pure um specification without any additional comments on that so i don't need to think about it

Danny Ryan: sure and uh light clients here i think he's he's one of the gatekeepers is that still makes sense for the engine api to live there

Lightclient: i definitely think the ultimate goal is to have it specified in the repo

but you know like like i said it is just a json spec and so as some of the work is still going into deciding exactly what these are named exactly what they look like it might make sense to open an issue in the repo or keep working against the hackmd until we get it to the place where it wants to be finalized and then we can convert it to the schema

Danny Ryan: okay i'm a bit hesitant to not get it to leave it in the heck md doc just because from prior experience there's just like a lack of depth on how people can engage with that

Lightclient: uh

so let's start how do you feel about an issue

yeah okay yeah i agree um also um the yeah the hakam did it it's difficult to follow if any update is uh there it's been updated like okay and

Danny Ryan: if it's an issue uh well again i i would probably argue moving towards code or towards something that you can follow a diff on sooner or later but if it's an issue i'd open like three issues for each subset of the engine api so that conversation can be split up a bit okay

uh anything else people would like to discuss debate or ruminate on today great let's plan on picking a fork epoch for altair in two weeks time um yes it was mikka uh and uh doing a lot of testing analysis between now and then so we can confidently move forward on that um and like i said i'll i'm going to share a document with you all later today about the beginnings of coordinating on the

next wave of devnets for the merge thanks everyone

talk to you all soon thanks ron bye bye thank you thank you bye bye [Music] [Music]

[Music] [Music] [Music] [Applause]

[Music] you

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.