All Core Devs - Consensus #50

Transcript

[Music] [Music] so [Music] [Music]

[Music] [Applause] [Music] structural differential fuzzer as a

result the fuzzing speed has dropped dramatically but that's expected uh but it's still going going quite well so we lost another magnitude in fuzzing speed unfortunately um but yeah we uh we started um poking our custom fuzzing engine so hoping to have that up running by next week apart from that i've been debugging a few discrepancies a few crashes um i have about three to four crashes that i'm currently investigating um so i'll be reaching out to the client teams that are affected uh most likely tomorrow yeah that's about it what is it about tekku that's slowing

things down oh it's just the the jvm really so instantiating the jvm so the way we're integrating tekku is by using jni's so essentially we're calling java functions from rust in which case you need to instantiate the jvm and that's just yeah the the fuzzing cycles are extremely slow but that's expected we've had the exact same issue when we first started playing with differential fuzzing a few months ago it is what it is when you have you know four different clients um go routines um a java virtual machine uh along with uh so nimbus and um and lighthouse it's it's it's expected

nothing really surprising there it's just just a bit annoying to instantiate the jvm each on each new um buzzing target or is that yeah so for each fuzzing target yeah totally not not for each fuzzing cycle so once you've instantiated it you're you're fine but um but yeah you do you do need to instantiate it for each uh each run um and it's um yeah that the fuzzings now take about 10 to 12 gigs of ram so it's not ideal but uh luckily we have that fuzzing infrastructure that we're leveraging for this so no issues really um it's just yeah just super slow cool any other questions from eddie

anything else here if we move on to client updates actually move on to zinken um good work on zinken i know there were um a little snafu on prism release stuff uh that caught the block explorer i think generally users had a positive experience and test net came out finalizing pretty well out the gate any more discussion or thoughts people want to have around cool zinken i just looked at thinking um

looks strong seems like four percent of validators faced since the last 12 hours turned their nodes off um i think more important to talk about is uh madasha wherein badasha stands um i've been kind of operating under the assumption that um because madasha is not v1o um fully v10 conformant and because uh we expect probably the vast majority of community users um to turn their nodes off when mainnet comes that it might be natural to sunset madasha in favor of a long-standing guano

uh test net that we can start either that is more largely controlled by us rather than the community for a longer term stability i don't expect users in the community to run two nodes in the long term they might pop in and out of test nets to test things we could do that call it two three weeks before mainnet launch we could also do that two three weeks after main at launch regardless i think in the time leading up to mainnet we should stand up a number of 1 0 test nets for ourselves to test all the machinery uh but i know that

there's some desire to maybe upgrade modasha to the lsv4 and this v51 to kind of test these components that load so i want to hear y'all's thoughts rather than just give my own thoughts on what to do with madonna hey preston here i just wanted to uh advocate for keeping medusa around after mainnet um the reason being is that uh like even though it's we might see a big drop off after a main net and it validates will have to be exited and all this stuff

we'll still have you know a month or two of syncing data that we can keep around um so as we're testing um new features and things for mainnet we can try it out in madasha and see you know okay we have all this data and have a you know a lead time on on maintenance so if there is an issue say after syncing um ten thousand epochs we would catch that earlier rather than um you know seeing in the main net so that i i think it's a it would be worthwhile to have some already existing long-running um testnet before we go to mainnet and keeping it around but yeah i do think it will be i agree that it will be kind of like kind of like we're seeing

now it'll be difficult to to keep it up if people aren't interested in it yeah i mean as madasha stands today i'm actually fine with having a stressful three weeks of um non-finality and seeing a leak i think that uh you know we are probably slightly bored with madasha too in the sense that uh stably it seems to run fine and now we get to put in a stressful scenario again probably not as stressful as what we saw a couple months ago but do other i think certainly there's there's two issues in keeping medusa around um it's that blsv4 upgrade which is

actually probably relatively easy uh but there's the disk u51 upgrade which is potentially non-trivial um do others have thoughts or opinions on keeping the dash around a preference one way or the other and uh thoughts on an in-flight upgrade of 505.1 on display i personally would like to upgrade with asia to basically v4 and this v5 just because as a client from clients point of view it's kind of hard to maintain two branch and one from adasha and then and then like one for the latest

yeah well once once we have a main net we will also probably not drop it but rather upgrade it so maybe it's a good good chance to test even more involved upgrades on a running network the harder of the upgrades is the disk v5 proto you um spent a little bit of time playing with that yesterday you want to chime in yeah so to smoothen the migration path from version 5.0 to 5.1 we can run this additional software that basically brings both versions at the same time with two identities then tries to bridge the data from one to the other

i've made a prototype of that i'm still trying to polish that's tested so that could help but it's only a tool on this migration path i don't think it's the like the core component the alternative being um at a certain point in time all nodes do a script migration from 5.0 to 51 format and don't have to maintain two protocols simultaneously and put a stronger assumption on a coordination point amongst all nodes is that viable as well okay we've heard from prism i'd like to

hear from if any other teams have uh some thoughts on this the one problem particular that we had we already upgraded to the spec 1.0 but uh since the epoch for it one voting parent was changed and this is uh compile time settings for us it's now a bit difficult to support because we have to go back to custom builds which are very descent and we prefer to ship a single binary for all users i see and continuously this company 5.1 is a compile-time settings for us

so supporting multiple networks it's a complication i think it's similar for us the less chess next we have to support the better i know age was uh keen on upgrading um seeing how we could potentially upgrade medasha to uh the 5.1 so i'm not sure if perhaps we could attempt that upgrade and can test that after that anything else here um we don't have

consensus we have um i think compiled on constants make it difficult because mainnet has slightly different constants we have a desire to maybe try and upgrade a flight and then can it and then we have a desire to keep it running so that we have a longer a test net with longer sync times uh come mainnet so we have something a little bit more hearty to prototype against um let's um open up an issue we can continue the conversation and try to figure something out maybe by monday almost a few years um i don't wanna i know that it's also balanced with uh

the more particular things we try to do with madasha the more of other things that we're not gonna get to before in that watch so i know it's a time balancing thing as well um okay anything else here before we move on uh again let's i'll open up an issue i'd like to try to continue the conversation and figure something out there early next week okay let us do um client updates you can start with lodestar hey uh so uh we participated in a little

in a small way on zinken we tried to run four four validators um and uh basically our status was that uh we had some incompatibility between our uh i think between the urls but that the validator was requesting um on and what the beacon node was serving so our validator uh is offline right now but um uh i guess the the good news is that our beacon node is and staying synced to the head at least in a read-only fashion so that's good news for us because this is kind of the first test net that we have you know not sunk on so um

feeling good about that moving forward we're gonna fix up the validator interaction we've been doing some things with the validator behind the scenes so it's a little bit i think we should be able to participate now we're we're keeping track of reorgs and uh new blocks coming in on on an epoch boundary to to keep our duties valid uh get the right uh duties uh which was plaguing us in the past so um we're basically just uh trying to get to a stable stable um point where we have a stable

validator stable beacon node and then we'll revisit like where we're gonna go after that so hopefully make a new cut a release of something stable in the next week or two and then look at what's next cool um is one of the reasons you're able to follow because of the lower load have you all tested on the dash recently i think the the the reason is because um we have it i think it's the uh signature policy for the new signature policy on pub sub a gossip sub like that we we don't have that merch into our master

branch but we were testing it out on uh zinken so i think our i think we just have like better gossip behavior like we're not getting kicked and we're not getting you know blacklisted and whatnot cool thank you cayman i'll see you numbers all right the primary focus for us has been the our outgoing audit which is now in its final stages most of the issues that have been posted are now addressed and now we are undergoing the review phase of the fixes our next step we plan to join the public attack next attack next

uh in other news we've upgraded this to 1.0 version of the spec as i mentioned and we are working towards creating binary releases of nimbus for linux mac and windows we'll be publishing them initially on our github repositories which i guess that's important news our github repositories have been renamed now the formally called new beacon chain repo is called nimbus dash e2 and the e1 project is called nimbus-e1 we implemented starting from weak subjectivity checkpoints and i'm curious to discuss with the other clients

how do you manage the history of deposits in this mode in the what we implemented it seems that a checkpoint is something that should store the entire state uh the entire history of deposits up to that point in order to be able to create new blocks and some other small pieces of data we've started testing interop starting doing interrupt tests for gossip system 1.1 in the multinet repo it's still not enabled in the views we do for the public testnets but this is hopefully going to be enabled soon furthermore there is ongoing work on

reducing the size of our database this is probably the most significant problem with an invoice right now and we are along with this work we are reporting further reducing the memory usage which is already quite good and that's it for me thank you um we can follow up after the call but were there does anybody have any quick comments for zory on um block production in this context that is um handling econ deposits uh once you've started from a receptivity state point all right let's handle it outside the

call thanks sorry um hi everyone uh from the techo side we've enabled moderately strict weak subjectivity handling for now we're just logging warnings if the client falls outside the weak subject of the period because we want to avoid abruptly changing sync behavior on mandala we've been addressing issues from our audit deprecating validator tools to trying to send like region register like deposits hoping that people will use launchpad instead removing ability to enable disabled p2p snappy compression via the cli uh like strictly we've implemented stricter handling on our gossip cues and uh added like ancestry checks to our

networking layer uh we've implemented the standard validator api enough that the tech evaluator can now optionally run in a separate process and use the standard api events we still have a few non-standard apis to convert over or this leave one our discrete 5.1 implementation is now merged to its own repo and is ready to be integrated as a taco we merged the pr which would reduce our native memory consumption by about half a gig uh we looked into 1.0 rc 0 changes we got a couple of networking details to implement plus the bls private key 0 changes but otherwise all reference tests are passing that's it excellent

thank you jim um prism hey guys so from prismatic sites um we are actually really excited to release our beta of xero software by early next week and this is exciting because of it allow us to remove a bunch of deprecated flaps and then we also clean up our chain we also clean up our general flaps and and also in the background we're working on the e2 apis data layer we're working on slashing protecting interchangeable format we're working on this the 5.1 to make sure that we have a

graceful upgrade path we're working on bos v4 uh and the uh peer scoring and we're also aligning with the spec version on 1.00 rc and and at the same time with with uh also release web ui so please please give that a try and keep us any feedback thank you cool thank you congrats on getting the uil lighthouse um so we had a pretty good uh zinken launch i guess uh pretty much like

everyone else uh went very smoothly i just checked the validator leaderboard i think uh nine of the ten validators are in the lighthouse just good to see i've also released a v 0.3.0 also known as king flippy nips uh it's a breaking release a whole bunch of changes were incorporated into this uh this release we have a new directory structure we've integrated the week's objectivity checkpoint um upnp support as well we have a stricture slashing protection mechanism and we pretty much we have all pretty much all the standard api implemented we're working towards full compliance there's only a

couple of endpoints left as danny mentioned earlier we're also passing the v1 release candidate zero spec tests which is uh which is good to see um we've made some great progress on our graphical user interface um hermann has authored the draft eip for a remote signer api so it's eip 3030 please take a look leave some feedback i know terence has been commenting on the on the pull request thanks for that and um our auditors have been uh working throughout these the external security assessment so 12bits and this authority have completed their reviews and ncc is still going so uh nothing really major came up of these

uh first two reviews and uh yeah those findings are currently being addressed and that's about it got it thank you okay um research updates anything i'll share here um and i can give an update on what we've been [Music] doing in terms of uh weak subjectivity or sorry not weak subjectivity uh data availability sampling um so basically we've been thinking um about the question of enough

how to simplify phase one and kind of make it as oriented toward like just doing um having to having data on chain and uh and not doing anything else which is realistically something that probably makes sense anyway because uh you know phase one does not support any kind of state execution um and uh but there were a lot of kind of com of complexities in it that were kind of designed around uh pushing pushing for a particular state execution path and so if we're going with the mindset what that name you know again if it's not it's not even clear like

exactly what's the or potentially if any uh kind of enshrined multi-shard as data execution is going to exist further down the line then it makes sense to have a phase one that's more kind of just doing the design around doing the thing that it's doing well which is basically having incentives over blobs of data and but if we want to have consensus over blobs of data then one of the things that we one of the benefits that we can immediately try to capture is uh reduce the level of dependence on committees right and the reason we want to i think reduced the level of dependence on committees is that committees are this kind of uncomfortable assumption where either

there's a hard two-thirds of them uh two-thirds minimum requirements in which case uh more than one-third off-line would cause the committees to stop working or it's it's a more kind of flexible and dynamic system but then in that case if you can just knock a bunch of nodes offline then an attacker with significantly less than one-third could potentially uh break a committee so that violates the assumptions too and so in the context of data sharding if we want to not fully rely on the community assumption then data availability ability sampling is basically the thing that we can do and if we want to do data availability sampling then

this requires a particular structure for clients that to be able to make queries for particular indices in short blocks uh and so we've been having a couple of uh of calls and uh uh just discussions uh around uh just like what's the exact um in a form what's the exact format and then what's the exact structure like what subnets would look like what they would do what they would do and so forth and it seems like we've been making quite a bit of progress yep thank you vitalik

um any questions for vitalik i expect some of those that have been working on this we'll probably release some write up soon for further discussion um regarding reforming the short transition by adding the shot transition by adding the short transition candidate um is there any plan of uh merging that pr or or are we waiting realistically realistically that that that pr is superseded by this work um some of the um the rewards accounting could potentially still be merged so uh this is true yeah i don't want to merge

that until we get a 1-0 out regardless got it um i mean it's really a question of your date availability sampling and you have potentially committees that can do proof of custody and they provide different guarantees and potentially have different latencies and how the consensus can rely upon them and we're trying to explore that design space right now right so like examples of uh things that could potentially be different would be like one is that we we don't even need a shard state as it or or even charge this like as a construction i mean we in short transitions

like we might need something that's much simpler but like there wouldn't be a lot of the state related things um instead it would just be like if we do have a uh have a committee on top of the data availability sampling then you would just have like some kind of progress tracking that waits until it hits 50 percent or whatever threshold we want um another example is a replacing shard block merkle roots with arcade commitments right how feasible or commitments in your opinion oh technically it's a very very feasible the applications on ethereum have been doing them already

for over a year um i mean it depends entirely on the size right i mean the main the only problem we have is computing the proofs when they become large that's the the right like we did the math for one megabyte right and like i recall it not being too bad yeah yeah so it scales with um linearly if you want to compute one proof and um n log n if you want to compute all the proofs but with a large constant in this case we'd also need a trusted setup right yes that's

that's one downside of kcg equipment i forget are there existing bls 1231 setups that we can build on top of or would we just have to make our own yeah the falcon one i think is the ds481 right and using that and then just like adding a hundred of us seems reasonable i mean data availability sampling has one like a few of n honesty assumptions already so oh like is adding another one of and what if analysis assumption really

breaking much i guess and then in the long term there's a there's other things that we can upgrade to as well okay um anything else on date availability sampling any other questions before we move on gotcha um other research items people would like to discuss i actually have an update from txrx great on yeah um our team undergoes some recognization

because we are losing our texan parts which is very sad yeah but the research keeps going um alex is continues his work on um by spectrum spider also he looks into transpiling the pi spec into rust because rust has a pretty suitable memory model for programming blockchains and with this like translation it will be possible to build a pure functional form of device pack um also we have recently started to work on withdrawals to ethane chart um and uh if one is two transition design and strategies um the research

regarding that is also in progress hopefully uh you'll soon see updates on the either search yeah that's that's all from outside got it thank you anything else before we move on um yeah then um one thing uh we have been looking into um different metrics uh from uh different uh clients uh in this case we have been looking into four clients and we're gonna try to look into a few more later um and we have um gathered a list of uh metrics that uh we believe kind of um

are i either identical between clients but they have different names and we have put all this into a spreadsheet and try to kind of match uh different you know metrics that are equivalent between different clients and it would be interesting to see if maybe the teams involved in developing could take a look at the spreadsheet i'm going to add it into the chat and and help us check whether you know these metrics uh are equivalent between clients because it would be very interesting to try to see um how different plans perform in different aspects and um so that is in a first step and perhaps in a second step we could

think about trying to make some kind of a standard so that all clients kind of use the same naming um for these metrics so that this much easier to compare them in the future so these are uh prometheus metrics that are recorded yes exactly uh yeah um i think there's been an idle attempt to try to standardize on some of them at some point but i don't know if that ever got pushed forward uh but yeah thanks for sharing the talk appreciate it thank you we can try to propose some kind of standard that it's going to be kind of similar based on on the names

that the different clients use and then we can we can submit this for um approval for different clients and see what they think about it cool um can you open up that doc i think that it's uh locked right now ah okay yeah i would open great um anything else here cool networking obviously we have this five one uh five one implementations

um i think are near complete and discussion as to what to do with nadasha um are there other networking items that people wanna talk about today um and moving on spec discussion um anything spec related um i'm capturing a few more complicated scenarios that will probably appear

in the test vectors um for v10 um i might release those very soon um but otherwise general stability um phase one stuff uh there is that that big pr that terence mentioned uh that has some reformulations of phase one and some reformulations of how words are counted um but that will stay as is at least until we release 1-0 and dig a little bit deeper on to date availability sampling thinking anything else here great um open discussion

anybody have anything okay um well in that case thank you talk to you all very soon um keep up good work thanks everyone thank you [Music] [Music]

[Music] [Music] [Music] [Applause]

[Music] you