All Core Devs - Consensus #95

Transcript

okay [Music] [Music] foreign [Music]

foreign [Music] [Music] foreign [Music] [Applause]

[Music] foreign [Music] foreign

Danny Ryan: welcome to the consensus layer call number 95 not number 99 which is what I had on the stream 10 minutes ago okay it's Issue 6 18. in the PM repo and we have a few things on the agenda today I do not believe this will be a terribly long call but you never know um we'll go over anything merge related got one thing there specifically but can open up for discussion then speech research spec Etc and open discussion closing remarks um on the merge fund before we get into the specific issue are there any updates

related to test Nets mainnet um or anything like that uh did we have a shadow Fork

Parithosh Jayanthi: hey yeah we the last time we had a shadow was last week so that was midnight Shadow from 12. um it went without a hitch which is great news we used um we use all the versions recommended by the EF blog post I have seen that some client teams have updates after that version and I think the blog post has been updated as well so shadowfall 13 will be using the latest releases of world patents and it should be happening

tomorrow

Danny Ryan: got it I do I do think you know we might consider maybe not testing the merge transition but um you know this being weekly being part of our build process or something as as people have new clients but I think that's probably a separate conversation um ultimate little Canary Zone okay anything uh any questions about the

shadow Forks any other comments about test Nets or preparations for mainnet

Parithosh Jayanthi: um I just wanted to put it out there again be headset we'd be deprecating can after the better tricks um and better checks happened successfully so I I guess so we haven't heard anything about anyone using can for any sort of testing so I'd like to officially deprecate it maybe tomorrow

Danny Ryan: that sounds good to me okay uh some other thumbs up from

the chat okay I do want to plug at start and we'll plug it at the end there is a merge Community call um just in case you're listening and you're gonna get bored and leave um there's a merch Community call tomorrow Trent um we'll actually try can you just give us the heads up on that right now for any of the listeners

Trent Van Epps: sure yeah you basically shared all the relevant information but it's same time as the old core Dev slot so 1400 UTC 10 a.m Eastern uh and it'll be similar format to previous calls where we'll just go over the latest on test Nets uh things that have happened any new

information from clients or researchers um and just yeah basically have an open forum for uh validators to ask questions community members to uh bring up any concerns that they might have but there's not much between well there's there's really nothing between now and the merge so it may not be that dense but we welcome anybody to join including researchers and client teams

Danny Ryan: great thank you Trent

Micah Zoltu: not just for validators any user who runs a node feel free to stop in and ask questions

Danny Ryan: sure yeah great okay let's

um Mikhail you have an item here on the updates on optimistic sync tests can you bring us updates

Mikhail Kalinin: yeah thanks Danny yeah so um Subway has been working on optimistic sync tests um basically this is an extension of the foxures test of the focus test format uh it's extended by adding a new payload info which provides the payload status uh the state is that El client is returning once receiving the payload so the idea when this payload is available uh the El mock that

is used by test Runner gets feed by this information and responds accordingly to the um to the tests to to the information I guess from from the test um and yeah that's how we can uh like have a sync and payload then it turns out to be invalid uh we can test um latest valid hash stuff and sophisticated scenarios with like multiple uh syncing um Forks when all of them becomes invalid at some point and yeah and the test client behavior in the deterministic way um we got initial feedback from

client teams on implementing these tests and one thing that has been discovered is that there is a valid flag on every block record on this test format and was like a bit of kind of a source of confusion what to use what's the right value for the valid black we seem to figure this out everyone can take a look at the most recent comments on the yeah on the VR just dropped in the chat so um yeah the problem with valid black is that what to do when the payload is sinking but it eventually becomes invalid uh so what's the right flag

where uh what the right value for this valid flag we decided that it's going to be true because uh the valid black should always be true we went on block doesn't return an error and when the payload has been seen is sinking when they said it is sinking this is the case when unblock doesn't return error unless a state transition fails other checks in the state transition so please take a look um I think we can move this from work in progress to like to actually the pr and yeah I would my preference is this to get nourished like on the next during next week probably

yeah next week is the merch week so yeah yeah just uh yeah engage in everyone every sale client to take a look at this it's gonna be part uh best see you soon

Danny Ryan: cool thank you Miguel and xiaoi for that um if you haven't taken a look uh please do and there's also attach in the header comment or uh some actual sample tests and build doctors that if you want to do an attempted integration you can see if there's any other um issues that arise when you when you do so

Mikhail Kalinin: any questions about this issue

also to yeah one thing I I missed and this tests are optional kind of optional so um yeah but it's definitely added to have them implemented uh and supported by CL clients but yeah as optimistic sync as like and um a size thing to the main consensus specs so we also try to make it like a sightseeing thing to the main the suit so all old old folk Choice tests uh will not be affected by this change

Danny Ryan: great

moving on other merge related discussion points for today when is it happening I don't know is is uh the soothsayer on the call no um you know I think we've been monitoring bordell we can share these links on YouTube um and I think the other alternative which is something of a linear regression that looks over the past three weeks I think the other interesting one the other interesting algorithm is just the kind of naive algorithm looking at the latest

head the naive algorithm accelerated and then slowed down bordell kind of stayed relatively stable as hash rate fluctuated um looks like you know 14th late UTC into early 15th but who knows um I will say there's a lot of misinformation around the win merge website uh it seems like they're counting down to the difficulty bomb which I don't really know what they're doing there which then estimates the date of 9 13 and so I think a lot of people think this is happening on the 13th but then if you look right below that they talk about TTD which happens a day later so uh do not be misinformed

I'm by this website okay anything else merge related excellent there is one other discussion point in the agenda today this is also by Mikhail and this is related to a PR in the beacon apis on checkpoint sync

Miguel Mommy

Mamy Ratsimbazafy: yes we are planning your release currently in testing for Nimbus that will be high importance with a couple of convenience um uh fixes to tell users that they are well connected with the ER client for example uh or to avoid uh the warning on this world Gaff that you're not connected to Nimbus and also um an important performance fix so you'll have all the details in the release notes and they should go in a couple of hours or tomorrow early morning

Danny Ryan: got it

Mikhail Kalinin: uh getting back to checkpoints in kpi uh there is a PR uh there is a discussion in the pr as well which is valuable uh to to read it gives more context and all um like um more and gives more details on this proposal and on the general checkpoint sync stuff um yeah one thing before and we go further is that there was a comment by Micah that if we have this API it should exist it should it should be run on a separate Port

um like as a separate piece of of the interface I completely agree with this but before we make this into the spec I would first like to just know the general opinion on this API whether we go in to implement it as proposed um or not like the general feeling and what client developers do think on that and related to like the feedback um that I'm trying to get here is that this API actually has basically two endpoints and it means that clients that does not it implies that clients that does not

uh yeah I aren't capable of starting from aren't capable of being bootstrapped from just the state and requires some other stuff like block that is in the state that is referenced in the state they will have to um be adopted for and being able being capable of starting from the state why I think so is because the purpose of this API is to keep it as much lean as possible to increase the adoption by state providers to get actually more State providers across the network who are willing to provide the state and if we

extend this by like one more or a couple of more endpoints uh this will I think it will significantly diminish the purpose of the whole proposal and as for me uh we bet to not accept it and you know just use the regular we can API for strapping the nodes so and that's like one of the things that I would like to figure out before we move on with this proposal so do people do client will risk think that it is possible for everyone to and it is possible and it is it does make sense to to start from from the state is it like difficult to be implemented uh clients

that are not yet supporting it

Jacek Sieka: I think I can speak to that for numbers at least um and I think the issue is basically that when you get a state without a block um you're kind of forced to connect to live b2p without having a block which means bringing up the network before you have a database basically whereas with the block you can already have like a full featured database exactly the same as you have when you start from Genesis because Genesis is unique in that you can

generate the block from the um from the state so it's not impossible of course it's inconvenient

Mikhail Kalinin: but why why do we need this block body at all I mean is it like a requirement on the P2P request stuff or whatever else

Jacek Sieka: um it's not a strict requirement it's more that much of the logic is built upon the block being there although we can pull similar information from the stage from the latest book header

uh it's just a good organization matter because we can assume that all non-finalized blocks are available to the client when it's starting to validate incoming blocks of whatever kind whether those be constant blocks or you know get locked get Beacon blocked by root requests and so on and um there's actually a similar issue facing light lines which is basically that uh clients that are snap sinking or there are checkpoints thinking this way they probably shouldn't be connecting to gossip in the same way but they should be making requests

from you know get blocked by root probably um and uh those are kind of Frankenstein clients that don't like until they have downloaded that block and sort of fully initialize themselves they're kind of Frankenstein Clans and I know that uh you know it might be difficult to maintain connectivity while you're in this Frankenstein State because full clients will not know exactly what's going on with them so so it's certainly not impossible but it's also a bit of work

Mikhail Kalinin: so it is more about like uh the inner

logical client which requires the block to be there not like the some kind of hard requirement on the P2P layer to be able to serve this block

Jacek Sieka: yeah I would call it the gray area of the ear to pair spec the peer-to-peer spec doesn't really consider it only considers full nodes right and there's a couple of other gray areas so this spec explicitly says that you're in order to be a participating client you have to serve blocks but obviously when you're checkpoint syncing you're not syncing block you're not providing blocks for sync which is also kind of a gray area in the

spec and we should probably address that at some point

other

Danny Ryan: other clients that also have this requirement that want to make a case I've been

Sean Anderson: well I'm not too familiar with the checkpoint sync the White House but from poking around a little bit it looks like we do use the state and the block

when we initialize it so might have a similar sort of requirement

Danny Ryan: but I believe does can Lighthouse pull down a week's subjectivity state currently and start or do they have to pull down both the state and a block

Sean Anderson: I had to spend a little bit more time looking to answer that but I can add some feedback on that yeah pull request to the Peking API repo

Danny Ryan: all right yeah that'd be good um I do I think some conformance here is

moderately high value um sooner rather than later so yeah if you can if you can do a little bit of diving and anyone else that is also unfamiliar with the exact requirements but um to chime in that'd be great I certainly agree um you know I I've had a feeling maybe this shouldn't be in the beacon apis spec repo I but I if it is to be actually specifying a separate port and thus like enabling being kind of a separate flag and um is the safe path and I I do think that that's a night a better compromise than trying to put it in another repo because it is an HTTP API on the beacon API on the beacon node

Jacek Sieka: I mean the other thing that I thought about that PR was a balancing thing like if you're providing a state already first of all we already have a URL for that so having another URL for the exact same thing isn't really strictly necessary you can still have an API that offers you know the current URL on the new part um the other thing is that the state is massive right so offering blocks as well at that point feels like a minor minor thing so we're going to standardize an API for um checkpoint syncing in in a way it feels like

also useful to offer up blocks because that enables clients to actually download the blocks as well from uh from that checkpoint provider and thus not burdening the peer-to-peer network with with that and at that point I would also use like the standard block s including that in you know minimal block sync supporting port

or check when things

Danny Ryan: so state is only the state access is only that's a debug namespace right

Jacek Sieka: yeah I guess

yeah yeah

Mikhail Kalinin: and yeah would it

make sense if we like uh if they checkpoints Inc API would provide like a block and state there in SSD encoded is it difficult to to make this bear out of I

don't know

Mamy Ratsimbazafy: so everyone is cheating and providing State and block in practice

Mikhail Kalinin: you mean that we already have it

Mamy Ratsimbazafy: yes

Mikhail Kalinin: yeah you mean that they provide an estate and block and like one response

Danny Ryan: yeah is that bundled

Jacek Sieka: no it's not bundled

Mikhail Kalinin: yeah yeah and we want this end point to to

return SSD encoded object so if we would like to you know provide an A A Blog as well so we can have this a new type of container which is basically very simple probably that's the other way

of making

Jacek Sieka: I mean we already have an API that Returns the block SSD encoded as well as making one or two requests doesn't matter it's a massive difference

Mikhail Kalinin: it doesn't make like a massive difference but if you have an end point that there could be first they could be a race racing issue when you're requesting the state and then you're requesting the the block the finalized

block um you will have to yeah I will have to I don't know what if like this this gets pruned by some provider uh so you're a bit between the two requests that you are making I I would prefer this to be one request and if we want a Blog on the state and they should rather be uh responded in one and they should just be bundled together in the response and these checkpoints in kpi could not only be implemented by clients and exposed as like a client um as a part of a client but it could

also be some I don't know very simple HTTP server that just you know um exposes these binaries and these binaries are being updated by by some client so the client doesn't uh isn't available from outside of the internet but this restricted HTTP service is that that's the other way to to to be to become a state provider and yeah in this case we we would like to have as less information required to be exposed as it's possible

Jacek Sieka: uh well I will mention the alternative

here which is basically we've talked to a couple of providers and they're actually open to serving error files as well linear files contain exactly what you're saying except that it's a full day of blocks instead of just one block in one state um so that's that's kind of that's a ball that's kind of rolling very slowly uh because we've been focusing on the merge but nonetheless it's it's also out there as an alternative

files

Mikhail Kalinin: they can be exposed via this API as well

Jacek Sieka: right I mean the information yeah I mean they're just dumb files right so you just you just

run a static HTTP server serving a directory basically and that's it

Mamy Ratsimbazafy: or BitTorrent I've put an example era server on the chat

Danny Ryan: thank you does yasik want to specify a PR that for the arrows on the API

repo

Jacek Sieka: sure

actually a good time because I was just Gathering my notes for that the other day so I post there's a new post on East research linking to all the relevant stuff um I can also put up there uh we connect apis back a consensus spec or something else

foreign [Music]

Danny Ryan: call I do ask teams to chime in on this issue um as well as

opening up the discussion of the error files um uh you know Simplicity is nice but if block is ultimately a blocker um I think adding another end pointer looking into bundling isn't too crazy but let's take it there um and if it's not decided by Devcon we will have to Hash it out in person great let me find my agenda it looks like anything else on just research specification

anything in that domain

Lightclient: um can I ask what people are thinking about the various ux PRS for improving the connectivity between el and cl we we currently have three proposals if you haven't been watching there's one that sets its default location to put the JWT on the file system then yasik had an idea that we should just expose the engine API on auth at uh or whenever it is bounds The Local Host because it's not accessible externally

and then Martin is also now proposed that um as an alternative to that we still have it off whenever it's bound to localhost we just use a set key and his motivation for that is that that doesn't allow anything from the browser to call and potentially manipulate the engine API so I'm curious if anyone has any thoughts or feels strongly about a certain direction

Danny Ryan: by the set key meaning it is predefined

Lightclient: predefined why does it either be empty all zeros I think the rationale Martin gave is that in the browser you can't

set the header field like that and so it's not possible to actually call the engine API whereas it's not clear if you can call out from a brow and someone's like correct me wrong it's not clear if you can call out from a browser and not have the origin header set because my original Proposal with yaw six proposal was to just in the El have the ability to block anything that's coming from An Origin difference than what we're hosting the engine API on but I'm not 100 certain it's possible in a browser to ever to ever call out without origin and so the simple thing is just to put a fixed key in and then it's just not

possible at all

I mean that Pig's redefined uh default kind of key that is known by every client comparable actually great

Jacek Sieka: I mean the fixed key option sounds good to me it's easy to do

Lightclient: okay anybody against the fixed key option

Danny Ryan: yeah I mean if we want to go the localhost path just if we think that's a sufficient patch sounds nice

Lightclient: you're saying without the fixed key

Danny Ryan: I'm

saying if the fixed key is a sufficient patch for the browser um I'm not a post right I do not have the ability to currently assess that

Lightclient: yeah that's what I'm trying to understand as well because I mean it doesn't like I would rather not have a fixed key like it feels a little bit like a hawk rather than a proper solution um fair enough okay I'll look into it a little bit more and I will um I'll drop some actual PR and I'll post some of the

consensus though yeah

Mikhail Kalinin: I'm just curious how it will be implemented and supported by yells so if you have a not so if you don't have uh sorry so here if you change the um the IP address uh for which connections are expected right so you have to it will create a jwtwt secret file as it currently is if nothing is specified then the default is like the localhost IP or the local host name and yeah this this default key right

Lightclient: right so if the day passed the

JWT CLI flag in then we'll do things the same way and if they don't pass it in we'll use the fixed key when they're about when they're binding the uh the engine API on

localhost

Mikhail Kalinin: cool and on the on CL side it will just try to use if if no keys if if no JW if secret is specified it will just use this fixed key right

Lightclient: yeah and I'm wondering if maybe it would make sense like even if the JWT key is passed to like try and fall back if it's

on localhost to try the fixed key and throw like a warning or something because right now like suppose the user is only setting the JWT Secret life of the CL now if they update their El client it'll use the default key and if they don't change the the the command line Arguments for their CL the CL will continue trying to access that old JWT and connect via that so that could be a consideration

Danny Ryan: okay I do

given there's many proposals I do suggest Maybe closing out some in favor of others refining one and then circulating it um just to reduce you know if there's key people to look at some of the proposals before you close them absolutely but just to reduce the overhead of getting a round of thumbs up I think honing on one

Lightclient: sounds good

Danny Ryan: thanks Matt anything else before closing remarks anything else technical okay uh Tim

has a late comment into the issue Tim

Tim Beiko: yes um so I I posted about this on on Discord yesterday and was curious to get kind of a side of the check from from folks here um basically everybody's been working really hard on the merge for a really long time um and there's kind of some uh pressure building up to discuss okay what happens in Shanghai from here and how do we sort all that out um it feels like you know given that we're about to have the merge um there's Defcon coming up soon and I it doesn't seem like people have like spent a lot of time or had a lot of

bandwidth to think through all the Shanghai stuff yet um it might make sense to like take a couple calls off uh so people can get a break uh recharge a bit and also take time in like a more async way to review all the different things for Shanghai and that we could uh you know have a call start again shortly after Defcon um and you know there were different like flavors of this some people said they really like having weekly calls and maybe we can do something that's like less formal in the party lounge or something like that um but generally curious what people think about taking you know three or four calls off uh once the merges happen and we've had like a call or two to debrief

um and maybe trying to set up a more async process to review the different things for Shanghai and three or four meaning the sum total consensus layer and all cordos yes correct

Danny Ryan: so like a month total give or take and that month includes Defcon which you know for the consensus layer calls I think we would have the one on the 22nd um not have the sixth and not have the 20th so those are the weeks straddling Devcon

Tim Beiko: yeah I think for all core devs we could probably skip the uh let me look at the numbers here uh we would have a call on the 15th assuming that the merge is not happening there we had them in time or we might have to merge on the call we'll

see how about we would skip the one on the 29th of September uh and then we would probably guess let's give the one on the foreign ly concur with Lucas's comments in r d server

Micah Zoltu: then I would rather just keep the pattern keep things regular and consistent

Jacek Sieka: I don't mind because I'm going on vacation right

Danny Ryan: I mean from a from a practical

standpoint on the consensus there call I believe people will begin to be traveling on the sixth yeah and I know a lot of people take the week off after Defcon so even those considerations um push me towards this

morning

Tim Beiko: yeah anyone else have strong opinions okay uh we okay so I guess yeah

Danny Ryan: I have heard some you know moderately not as strong but lots of

thumbs UPS um outside of this call just to add that weight

Tim Beiko: yeah and I guess obviously this is like the CL call So like um you know if the idea of just like skipping the call before and after Devcon um makes sense to the group here we can discuss you know what we do with all core devs next week on the alcore dev's call um but yeah and and there will be no awkward that's called a Defcon so it's like even just skipping those two CL calls plus awkward as a Devcon basically gives you like a three-week gap between um between the

call soon

Danny Ryan: so cool thanks Tim

okay anything else for today anything else on this call before the merge cool okay well happy merge everyone talk to

y'all soon [Music] [Music]

foreign [Music] foreign

Speaker names adapted from ethereum/pm, CC BY-SA 3.0.