All Core Devs - Consensus #13
All Core Devs - Consensus · 28 Feb 2019 · 1:55:14
Eth2 client teams and researchers cover testing, client and research updates, then discuss the phase 0 wire protocol, serialization and spec questions.
Speakers
Topics
From All Core Devs - Consensus, hosted by Ethereum Foundation.
Transcript
[Music] [Music] [Music] okay I just transferred the video to
start on the chat on YouTube let me pull up the chat and see if they can hear us okay cool I assume they're gonna let us know on the chat soon if they can hear us which I think they can okay everyone has the agenda I'll share that in the chat now and we will get
started okay so we will start as we did last week with testing updates I can get that started I believe since the last call that I shared a very pretty simplified for choice tests it's very abstract in the sense that it's not really talking about a lot of the data structures we have it's more just like building a block tree saying what the recent votes are on that block tree and saying what the head is it seems like everyone that's taking a look at that has said that that more abstract version is better the reason about and that won't be too much of an issue to build within clients so I think we're gonna
move forward with that which is very good I shared some other ideas on state tests which I know there are some pending comments that I have not responded to yet but they don't wants to chat about that now the idea is specify a state the starting state specify blocks to transition and then specify a resulting state or a resulting subset of the state if we're just testing some particular portion of it I know Yannick have seen cut off then oh can you hear me now so you hear me yeah you okay sorry that I was vocal mommy cool anyway the state
test those are the state testing for choice s and they are big for interoperability so if you haven't taken a look at those take a look and comment and Yannick I know you had some comments and we can talk about them now or I can hit them offline and we can focus on that I think both fine that's fine great and then beyond that does anybody have any updates on testing so we've started playing with prismatic s-- chain chain chain test format just to try and you know be like other people just
something we're doing what came for the state-transition test when they come great and what is the format of the chain start test the play where do you ever like you can sure yeah I think okay great is there any plan on putting putting that test in the e2 reference test repo yeah you know yep okay sorry yeah we certainly do that I can't open the pr in the next couple of hours right on okay
cuz that the only thing I was worried about is like it is GPL and um like I don't I'm not a lawyer or whatever but like I was worried about any kind of copyleft its location it's we're at YouTube so yeah I mean either we can discuss farther offline dude yeah cool yeah and I'll take a look at those tests offline as well thanks it probably needs to be extended of it before it's you like particularly useful for the f-test repo I think it's a little bit more like internal sanity testing at this point I see okay great any other yeah yeah I've also discussed with Zach Cole for my
white blocks this morning so of currently open-sourcing part of their tests and simulation and I will link to history poll where he has some kind of structure to lunch the daughters of Ohio's clients and he has example configuration for Ephraim he owes Bitcoin and so to create a network a test network so I'm showing that in two seconds oh yeah I know he's been working on getting that open sources exciting quick one from me Daniel just a heads up
the the BLS reference test in the repo are somewhat broken I think the serialization format is incorrect so I put added to an issue that's in there what I found but I'm not making a big deal out of it because I believe the bill that stuff is all you can change in any case right and I had to do with the compression flag yeah the compression flags and the imaginary and real ordering is incorrect as far as I can tell compared to the spec the respect could be clearer on this matter okay great one of us will take a look at that today Thanks okay anything else on testing forward
move on great now we're gonna click the run through client updates how about Pegasus starts alright this is Steven Trevor from Pegasus we made quite a bit of progress over the last couple weeks we're in the process of upgrading to the 0.4 version of this back we we have simulated blocks from my Mach network adapter kind of running through the client so we are running it closed looping related know that executing state-transition and at the moment executing a simple fork choice rule but we're ready to plug in LMD ghost been has been instrumental for helping us get
VLS set up and running like you mentioned we're still waiting on some integration tests for it but we have BLS with Milagro we're for the working for the most part and couple of our next steps you're just working with the other research team with the Pegasus for narrating handle which is their signature aggregation library and then perhaps making our state transition core pluggable so we can use their with their Wittgenstein consensus protocol simulation cool ok progress how about Trinity week I think we are still working on sync with the latest
spec and most of all so part of the research team so we try to sync the latest one version instead of the previous snapshot releases and on the network side we are test named a star after that p2p network with our current Trinity technical technology stake but also Kevin our team working on somebody with the golden pupa demon and there is PR and it's working
for us right now that's our update thank you great thank you how about prismatic guys we've been doing a lot of progress on getting our real run Tempe so we've been we're pretty much complete with B 0.4 Saturn so small bug fixes and just really trying to get a simple test out of like eight validators in the beacon node running just to make sure that we can advance the chain indefinitely I think we'd advise everyone to be super careful when it comes to committees like fighting committees optimistically if you're gonna do that at the start of an epoch there's a lot of weird boundary condition so there are lot of parts in the in the spec that specify like
currently Park next epoch previous epoch so if you're like it's really easy to run into really scary bugs that just don't like that you just crash at specific be parks so that's something we've been running into but aside from that we've been just optimizing that adding more caching layers to items in the repo caddy things to move we finished implementing initial sync of our beacon nodes to say block since last finalize ones and mic state but aside from that or 8-valve bitters are all like making money efficiently rewards are being applied we're never more than 40 bucks is finality so things are looking good we've been just having some trouble with like assignment fetching and the sea changing at some point so Danny helped
us out a lot but yeah impact over the next few weeks we're just gonna be prepping more for that like some sort of patient really yeah great and I was happy to work with you all to find some those folks yesterday which there's a couple PRS and there's another pending but that we found that um I'm gonna go fix up for but thank you very exciting how about Nimbus hi so smile out the prismatic a lot of good work towards a simple test net and we come true we also had a lot of epochs and the slot issues that we debunked and that we are
still debugging in terms of spec like we said two weeks ago we are targeting zero point three instead of zero point four to reduce the impact of changes and we now have full same initial implementation for become chain and the block pool to deal with missing blocks to requests to other nodes in terms of protocols we have been working on our LP x on issues that benefits both f1 and f2 and discovery and regarding Lippe to B we also make make progress in parallel
to that regarding cryptography because it relies on erase a nice sexy and eg two five five one nine curves for various handshakes and we implemented all that so we will be able to use fully p2p in the future and besides we have been working on the documentation generator so that we can have proper documentation when it's time to release that so that the public can play with it in terms of for to do we have pls multi
verify in the pipeline and for the test net the main pieces for trace and also like I said in agenda and shouting I will be doing a presentation on Tuesday at each DC on the Ephrem tests and simulation so I will be focusing on testing because everyone is still working towards the test net and for simulation we still have a lot of projects we can use like what white blocks has been doing what consensus has been going with
Wittgenstein what layer was being presenting so far Leo from the personal a supercomputing and of course they can also play with the proof of concept that vitalik did in ephraim research trip oh one thing that would be super helpful is I will refer to all implemental projects so if you kinda had status on you rate me like if you are dating in station if you had some installation procedures that would be like helpful thank you okay thank you
yeah we much like a lot of the other guys we had a lot of progress we ended up not even hacking Andy thunderin just kind of working on our stuff we got SS Zed fly up to speed we pretty much done with BLS in regards to porting over the using lozen so we were using hiromi's library running done was um kind of we've had to make a couple changes to it just make it work for us and that's almost done we have us is that important arch and we're using and we think we have a branch running with us as well we finished up all the state transitions we found a few bugs in epoch and we've been
kind of working through those and thanks for your help there the other day Danny point me that if you yesterday that was really helpful and we're pretty much just like architecting out the last little bits of what we need to kind of what like the plumbing will look like when we go and hook up everything um because we're prepping for simulations on our end hopefully in the next like week or two we can start like finding getting it going and just like kind of breaking things and fixing whatever we have issues on I'll say and I don't know where my care is that with lip p2p but I know she's almost done according over adding some
of the functionality there cool Thank You harmony we have added support of configuration files so it's not possible to pass didn't change parameters in the yellow file was also edit the command line interface for simulator then we've been playing around around the general simulation scenario with just for validators and found the number of box
and the implementation and fixed them also have found a couple of issues in the spec but our spec is not up to date and those issues or the fixed inlay and latest version so now one of one of the things we are working at the moment is they into the 0.4 stack version also we merged a batch to license finally so now we're open source and the next steps regard with regard to simulator is to
try some unstable validators behavior when the stations are lost or blocks are not produced in some spots but to start working on that we need to finish current and we need to make the current general scenario work well so yeah there's seems to be all four of us great thank you how about parity so for our so we learned some really simple test nets for the just for the Kasparov FG and it helped us to sports and the
performance issues somehow term is we can fix so we managed to right now catch around 10 cloud per second but our analysis is most of the time spent importing blog is for the PRS back verification still also we figured out it's not super slow that is still gravity Li not very fast so but in like black fur for propriety assume for a po8 highs net we can get like over a
thousand flocks per second but so we are a little bit shocked that we can only to ten blocks per second but I guess it would be fun for for real men that's because we will have is not a purely network and we have a lot of meditators anyway but this is something we spend some time looking to and right now our serger system run timers to updating our spec to the to the newest version so so we got standard around implementation of the committee and shuttling but we haven't integrated into the actual stage
as a function and yeah I think we will also start looking to those tests big cards which we haven't before so that's for us okay thank you and I think although we will get some speed ups in the signature verification over time there is going to be some like order of magnitude difference on being able to process blocks because of the the simple operation of checking the difficulty and with work versus the signatures great yes Dean so I updated to spec 0.4 version
currently working on a bit of refactoring so we can clean up the spec or the implementation so it's no longer one-to-one coffee and then still looking into networking we found someone who already did live P 2p headers in Swift so I'm gonna try and see if that works for us so we can use that then we finally managed to get the a list of compiling properly but we're having a few issues with the bridging header so we might move to a new BLS library which prismatic labs recommended to us but we're not fully sure on that yet cool thank you Oh lighthouse Hey yeah so like everyone else we've been doing some
internal updates optimizations in particular we've been a caching committee info which has got us to to do some PE processing with 16,000 validators in about 1 millisecond with pre-built caches we're trying to work towards building you know sub 1 second block imports we've also been building the fork chair fork choice test framework that you were talking about earlier the start Danny so we have like a working version of that which will be modified once the most actual framework comes out we've also been starting to build out our core network syncing and service
infrastructure which we then have to link into our gossip rust Lib p2p gossip sub implementation which also probably needs a little bit of work for chucking that in and we also have a rust developer starting next week to work with us great did you did the gossip sub stuff get merged into the Lupita Lupita P it's not merged yet it's yes it's in the process there's a few still things have to update and fix with that I yeah I'll be doing that as we as we merging into a lighthouse yeah yeah there's a there's a few things that the way that
the the rust lupita be set up is very different to the go-live p2p which makes things that are typically okay did I miss anyone I don't think so okay cool I wanted to do a couple of quick intros when we want to research Mike from lead peter peas here i know a number of you met him at DEFCON he works with protocol labs you want to do a quick intro Mike yeah sure odd yeah my name is Mike gelser I'm from protocol labs I work on the p2p along with Raul who's also on this call and just just joining to get a better sort of
understanding of kind of what's going on with the 2.0 implementation so I probably won't say a lot but we'll listen carefully cool and Matt Matt slipper is here who has done some work on the networking spec recently Matt you wanna give yourself a intro you know I'm in Super GT Oken were on a native consultancy I wrote the phase zero wire protocol Andy stay to be 2.0 report the amine and Dan so really looking forward to helping out great thanks glad to have you cool anyone else that's new that wants to do an intro that I missed
great we will move on to research updates ah sister I'm happy to start all right so created a a meta issue on github called miscellaneous we can change changes take for so basically keep track of some of the remaining changes that we'd like to make for phase zero hopefully that would be the last such meta issue so it's partly for visibility but also just for keeping track of things I've started
thinking a little bit more about moving towards and an executable spec as opposed to a spec wave like human words so I think example specs will help kind of converge everyone and help with testing so one of the things I did yesterday was submit this ten Yvonne T for such easy steps back button and go and Diedrich AKA protocol vendor seems to be working on that which is fantastic and you know there's this kind of constraints on the line count and so
this kind of force is a simplification mindset so it's possible that you know we'll have inbound simplifications to the spec thanks to that I mean it seems that protocol lambda has already found ways to simplify the presentation one of the topics I want to bring up again is the sha-256 versus catch X 56 design decision basically to simplify this on the pro side of things there's the interoperability because many different watching projects are moving to watch a t56 and then on the con side of things there's security concerns so one of the
things that I realized recently is that it's quite likely that the in the context of the standardization for BLS you know for the hash to Jiwon has Jeter that we would need standardization on the hash function so if if we do want a standardization of the BLS then it would definitely helps you have strata 56 here and I guess one thing to mention is that kid check three five six will likely never be a standard that is adopted by the other blockchains
because it kind of missed the boat with Shastri so in that sense connect five six is a bit of a dead end on the security side of things i think the the main thing we want to make sure is that there is no like deal breaker and one of the pieces of good news here is that our hash function that we choose today to last decades necessarily it's possible that you know ten years is a sufficient kind of lifetime that we're looking to to have so one of the things that we're looking to do is basically ask a stand
when they what he thinks about the security of sha-256 the two things that are known is one is this length extension attack which is it seems mostly in annoyance because in some cases you need to put mitigations to remove that attack and then the other thing is that there's these kind of mostly academic security reductions so like small attacks on sha-256 which are in no way practical but kind of not ideal and an NK check - five six don't have them so I last unbeli and and see what he says here so I think the the
design team is kind of moving towards the opinion that if if there's no real deal breaker on a security standpoint then we would favor interoperability both at the the basic hatch level but also because it has implications for things like like the elastic tension and the final thing that we discovered very recently is that if two lifelines might be more valuable than we thought partly because there's just essentially a denial-of-service attack on the on the
fork choice rule and so we so the attack is basically that an attacker could try and have multiple Forks and force all the life of the four nodes to kind of evaluate every single fork and each fork is has kind of a minimum amount of what they need to do because you need to apply the epic transitions at the very least and so one way to get around this potential denial service attack is to use like clients for to evaluate the fortress rule so we you know I might be a little bit more time on my clients
great so on the sha-256 one we just want to give you an update on where we're at and to like I know some of you weighed in a little bit on that but we want to want to make sure to get kind of the wide set of opinions on that or at least technical perspectives so I know there's some some people drop some stuff in that issue already if you have something to say right now please say it and if not you can say it all flying on that issue so I have one question would you said are you considering sha-3 at all as a candidate because I think it's much more widely available in in terms of implementations so and I guess it might have you know one two more years additional life span
over sha-256 because it was invented a bit later so one major kind of desideratum for hash functions is unfortunately a easy executive ility inside of the existing evil 1.0 chain because the Abbasid contractor needs to be able to generate the merkel branches so we haven't thought of like shot to be the standardized version we have brought a thought of Blake because I think Z cash was using and I think they and they move to some peterson thing a hash based thing where they use blade you might still be using blade for some some of it and the main problem there is that
basically there's not a and a remotely gas efficient way to do it it's not an eighth one chain which could end up like you know so delaying launch by months yeah but there's like there's no real problem with just adding a new primitive for you know like any hash function so yeah i guess that this and actually if we are going to go with a kind of more progressive hash function then Blake might even be the best one because I know like super already wrote on the IP and I think might have even contributed some codes worded so so basically I mean just you know like adding primitives like that it doesn't like decrease the security of the EVM at all or it's you know it's not a contentious issue or anything like that it's just you know if
any particular hash function is needed it could be added I do agree though that users might actually take some time because of the you know how slow the whole four processes and everything right and one thing to highlight is that sha-256 other than just a couple of blocks is really becoming a standard that people are beginning to use so one of the arguments is this interoperability thing whereas Shaw 3 even though yes it's more standard across the industry the wide industry is not necessarily being picked up by block chains to be used we know what boy who is B what Blake is
being used by by the way polkadot hmm okay Dodd hmm but I should talk to poke it out because you know the whole selling point is interoperability and so it seems like a strange choice of hash function they might be depending on the speed up game I dropped into the discussion the multi hash from protocol labs which kind of works around all of this simply quite well received does that being considered what multi hash just means like do you have a choice of what
hash function to use and that's not really something that we can do because of watch in requires ever like everyone to use the same function or at least if like people use multiple hash functions and they will all need to be calculated yeah and yeah I mean it would have to be multi hash is basically a way to label a hash with the hash function that it created that value so it's you know it's useful sometimes but you know and specifically we're still we'd still need a code even if we wanted to support multiple hashes there's still need to call us upon the hash that will be used inside this deposit contract that is in for the navier-- all right but it doesn't have to be the same as
everything else that could be flagged by the EVM as a shot to 5k track two five six as it is and we can just use something else for everything else we do and as long as we label our outgoing hashes I mean everyone can calculate according to the right hash what the state route is depending on what they've received I guess like in terms of interoperability like what blockchain or even considering that could conceivably have an ease to light client inside of it like the main target that I can that I can think of is basically like each one and I think if it's to be used as a pair of channel polka dot that they
would need some sort of light client and maybe early in cosmos okay do we know what the end ins this cosmos years hmm because like I guess the other considerations here is that like for any chain that's using like any of these was some derivatives and maybe they're fast that any of hash functions are fast enough to add to reasonably execute directly I don't think cosmos has enshrine VM there's a distinction between the VM Dickinson say oh we're right so what's cosmos like the way they do
interoperability is you they require the know the the nodes to also like to run specific exhauster which is basically a note of the other chain or something or something if that's the case then like it doesn't matter at all okay so I'll just run you know like our software on the so even if we if we decide to go on like a self-describing hash to come all the ash we still need to decide on some subset of hashes see is right yeah regarding which - I think it was supposed to be a way to answer a great
ability like we say okay we use multi ash but we only use Schaffer II or shadow or Blake but if like five years from now we want to upgrade as a hash for whatever reason it's easier when we are already using multi ash because it takes care of versioning and other things like that which is also an issue in SSD for example I don't don't be too hard because everything that is signed is like for example right like there is already this signature domain which represent and that's ignition or oh man
encodes the fork ideas so there's already a kind of standardized mechanism for a basically determining kind of which but like or learn which for camera objects a came from and so if you wants to just like flip the hash function from one from one to another add some kind of what you could do that well I guess the other the other issue is that it doesn't really make sense to be able like there's the place where multi hash makes sense is when you wants to be you know
be able to support objects that other people create with different hash functions but here we're talking about blocks and if you have to kind of be open to the possibility of a shot to g6 blog and of lake block and the catch a clock and MMC block and then what that basically means is that you actually have to calculate like keep all four state routes calculated for every block so which increases your work by a factor of four and then so that you can verify like any one of them you would need to but it solves the well it could go some way to solving the interoperability with polka dot for example well no like I feel like we've been using the word
interoperability without really guy knows thinking you know where the interoperability issue is because I don't feel like the interoperability issue exists because we don't have like EFS compatible like free prefixes somewhere like so for example the interoperability between e2 and e1 like the only issue there is that there is an ether one currently only has an entire of efficient support for sha-3 and and sha-256 right or it can't shout - of g6 and sha-256 well yeah it's like if we add one more precompile and i think like felix is right that honestly it a hash function
is pretty much the safest freaking pile that we could add given that it's just copy-paste it will expire in repointing a library then like regretted and one or into the mix but and then say if to to aetherium classic if like they wanted to that and they then they would need to be at the same precompile unless we either you charge 86 and catch up to 56 insider and then once we start talking about other chance and it's a matter of like what scripting language do they have and if they're scripting language supports wasum then like maybe actually our hash all our hash functions can be executed and wasn't fast enough what's the resistance to using the multi
hash format like I get that there's another workaround js it just seems that Italy it's like it's just a little bit more mad is basically like having one of the really nice aesthetic things that we have in the current spec is that's pretty much everything is a power of two and once you start adding prefixes of things that are 32 bytes then they're either bowling all the way up to 64 bytes or they're not a power so you do and and then you have to start like we have to start like changing around the lengths of all the different types and it starts getting complicated you know also in general like victaulic also kind of explained it earlier the the main
thing about multi hash is that it's useful if you are dealing with with with the situation where people actually want to use many different hash functions for different purposes but like in terms of you know if they're in consensus there's really only one way in which objects can be produced and hashed so it doesn't really matter so much you know to you know support arbitrarily many hash functions and being able to you know tell them apart right and to to be interoperable with aetherium is to just support and know that there's a hash used on aetherium and use that rather than having to like arbitrarily be able
to like read it on the fly yeah I think having a single hash is simpler it's just that we don't want to paint ourself in a corner in the future so even if we don't use multi hush just make sure that if we need to change ash function in the future for stocks for example we are able to do so without reengineering everything yeah I think we are and it's like and I'll describe again like how that would happen right so the like every single object that so first of all if we have a hard work to change the hash function from function one phone
from let's say for example from like md5 to sha-1 just a year's totally stupid examples then up until block 1 million everyone is like happily turning along and using md5 forth for their hash and then after one let's say 1 million is 2 4 o'clock any book at 1 million there would be a state-transition added the door that basically says well not even a state-transition it would just say oh from now on we're checking the state routes and we're checking blog roots and everything using sha-1 instead of using md5 and so clients what needs to kind of very quickly burn through basically rehashing the entire state which they
could probably do and like something like half a minute and that would be kind of temporarily disruptive but it's only one time and then from zero on any blocks after the transition would be they would just be hash with sha-1 the one corner case that you'd have to cover is if in an test station got created of a slot before block 1 million and it got included after 1 million then basically even though it gets included in the kind of kochava an era would still be using the md5 hash function but there already is infrastructure for handling that case which is basically a big like anything which is the BLS
signed has this domain attached and which basically includes that what which includes which orchids from and so you would be able to just go along with adding two ends use the hash function based the hash function based off of what the fork domain is that got specified so it's actually pretty low complexity okay to recap before we move on we're going to get kind of a security briefing on sha-256 there's seems like there's a little bit of a motivation to try to get Blake into eath one I don't think that getting
Blake in the eath one within this calendar year is I mean maybe there will be another Fork at the end of this calendar year so if but anyway counting on Blake being in each one seems like not something we should do for this year and then even if we are to support multiple hashes we need to support one of the hashes inside of these one which is shot 256 or character 256 for handling its deposits and for handling some of the fun things we want to do by exposing the eath one of the e2 savory anyone Multi hash seems to me like it doesn't bias that much but
we can hand we can continue to talk about this offline great let's move on more research updates we're still in research Vitalik you have anything you want update us on legs yeah actually so um things I've been working on first of all there is phase one so I wrote up a couple of an updated PR for the group of custody game which is number 682 and the main thing there is that the game is basically the same as it was before except with a couple of fixes but I managed to move the state objects they kind of outside of the validator record
so that will be a record keep records can be kind of perfectly fixed sized again so yay so with these one there's an you can like click on phase one directly from begun of me and aetherium 2.0 specification section the one PR that hasn't mean 682 hasn't really been merged in yet but honestly probably probably should be given that its thing I knew in with a bit of feedback you can like review to be like as mature as the rest as the rest of the document but I think in general there's just to go through it a bit there's two major major
sections to do it where the first section is shard chains and crossbone thena which basically describes how shard chains work how person is that committees work who can create shard blocks what the shard state transition looks like and also what the data is that should go into this short cross linked data BrewDog field which we're currently leaving empty during a phase one we're sorry during phase zero and we also talked about this before chest rule for shorter blocks and then the big section two is updates to the beacon chain and that basically adds the proof of custody game both not a one round
interactive branch challenges and multi am around interactive proof of custody big challenges so there um in terms of what what still could be done for phase one I opened up a checklist which is issue number 686 so there's one two three four seven items there as for the first is basically like is are they're kind of substantial simplifications or substantive that you we can do the crew of custody game might be possible I've tried for a week and haven't really found any yet number two is grammar phoenixes and edits number three is naming changes
economic review is number four which is basically just verifying that the rewards and penalty is kind of makes sense and you what they should be doing number five is the kind of multi-branch number five and six are basically optimizations that say in that allow us to and if reused Myrtle become a large part of merkel branches and reuse at the station data to kind of challenge for much larger pieces of data at the same time with relatively low and efficiency losses so like on net it's a potentially huge efficiency game and then number seven is another one of those kind of very separated out farmable research problems it's kinda it
feels kind of very similar to the design and design a shuffling function problem except this time it's design and mixing function where the goal is to be m friendly to us multi-party computation so those are kind of phase money things people or and have encouraged to read encouraged to comments and so forth beast who i wrote a document on that so basically issue number seven zero two which is kind of much more preliminary exploratory which is things that we even need to just decide on and decide for Phase two and I have a bunch of items there another thing that I mentioned is
coccyx and I started a brief discussion on how he was somewhat fit Ian and I had and maybe what the US an interface would look like so that's like a discussion that's worth starting to participate in I guess beyond phase two I also brought up a document where I made my proposal for what CBC fi machine what might look like much more concrete so that's number seven oh one of people wants to look through that they can feel free to there's also smaller improvements that do you figure it out so one of them is
where is it now a number 685 which has changed how about leader balances are stored so that's basically it it's there to provide two benefits one of them is reducing the hashing that needs to be done so and the second is to reduce the size of a like playing proof by somewhere between a thorough 1/3 and 1/2 the one other thing that I calculated is also just like the amount of hashing that needs to be that that needs to be done every epoch and right now like if we include this optimization then the
amount of hashing that needs to be done every epoch is 32 by 64 bytes per validator for the validator balances when because 32 and then multiplied by to visit Samer code tria plus 45 bytes per validator to I compute the shuffling function and the reason why it's 45 sorry not not 45 not bytes it's 64 bits per validator and then 40 45 bits per validator the reason why it's 45 and not 90 is because really in every round you only needs to compute half of the possible values so in total that gets us to about a honey then about a hundred and nine which help could help figure out and figure out like basically
whether or not the current design is reasonable in terms of how much work it forces on validators mm-hmm that's probably yeah those are the big thing big things at this point and then also the other thing in terms of an executable spec I'll probably make some pr's to try so I did an experiment for phase one which is to try to write phase one much more heavily in service of Python codes and in terms that in terms of like English bullet points and it seems to have worked well and I think we can make some PRS it's her in the face
like that's her Fieros back into something closer to that so because I kind of keep it exactly the same in terms of meeting meaning but write it out in terms of code and that would I think improve clarity and probably also also make it more and more executable which should help with testing and finding bugs and so forth any questions for italic for move on great so the phase one stuff is beginning enter into that kind of iterative design process like we had in
the phase zero so if you want to get your hands dirty and take a look at it and even write some code know that although the core functionality is there that it's gonna be continually kind of reworked and ironed out we protected well protected say not very good made better anybody from the Pegasus team wanna give us a research update yes so like Steven already mentioned we have discussion with Artemis team on how to integrate our signature regression framework Intel Artemis and we are also testing the quick transport protocol and we are adding new functionality like blog listing buzz internals and that's
how most of the team is on holidays great thank you is that handle yes is handle leery have a brief update for us yeah not much update just a quick comment I hadn't i concurred with Ken and Jake from the foundation as week just an update to have a quick update on on the progress on the on the grant and a simulator and they advised me to write a blog post showcasing simulator and has been done so if anybody has liked
ideas of something that you would like to see in this post or comments or something that this content means because I will do bike and it's in the next couple weeks so yeah okay great that sounds awesome and if you want to share a draft with some of us for review before you publish we're happy to take a look great thank you any other research updates for a move on um I had the question for Justine is there any like github issues about the fork choice does ha'tak yeah will write up something okay thanks
great okay one thing I wanted to kind of quickly bring up is like in terms of just efficiency avec like execution it would be like are we at the point where clients can give concrete numbers to like basically the goal that I think it that we're going for is that it should be pause at least possible to execute especially the epochs they transition within Leica less time and hopefully considerably less than once a lot even at the end of maximum four million validator level so do we have I kind of
statistics on executing that executing this transition with that number of elevators and if not kind of how far away are we from my actually being able to come up with numbers yeah we've been playing with numbers quite a bit um that seems reasonable the thing that's really slowing me down at the moment is the is a shuffling it seems like the permitted index is a thousand times slower than the previous shuffle interesting yeah so that's only you have executed the naive way right actually shall we do have that piece of the the piece of optimized code that I
came up with oh yeah well post it yeah it's always important to remember that what goes in the spec go is it like is often not an efficient implementation and in this case that's definitely true so you definitely like the code the optimal implementation is like something like over 100 sometimes possibly like faster than just the naive way of doing it separately for every index and as Diane mentioned earlier in this call I think even the optimized implementation is suboptimal by like up to a factor of two in fast languages the reason being
that you can only run OAC it um 2.0 specs okay you know the reason basically being a being that instead of hashing through the entire the entire kind of list of validator indices in every round you would only actually eats the hash through kind of the half that he is a possible and kind of maximum of like X and given and hit minus X for any round and so you actually can make cut the hat cut the
hashing down even for even further by a factor of two but you know otherwise that's something that hmm I wonder if this like shuffle code should maybe equally have somewhere as it's somewhere else in appendix yeah it would be handy to see the optimized stuff because the like I know that the spec is not supposed to be optimized it mostly you can figure out it's just normal programming stuff for that that's roughly an algorithm is pretty intense so if someone's already work through it would be pretty cool to share yeah yeah yeah so we also had the this issue on the the Nimbus team so we first used the naive way and we couldn't do the epic
changes during the time for one slot but we re implemented a shuffling using proto lambda implementation which is ago reported up to him and it worked quite well though we still had issues with timing so in the end we compiled with optimization for the slow paths to work around that but we will do optimization later but we still we already have someone looking into like parts that could be optimized do you have like concrete numbers for how long it takes to shuffle
like a long list of someone like a concrete I've been doing 16 K validators with the old shuffling the Fischer Yates it was one millisecond okay and with the new one which was so the new one that's like slightly optimized where you don't do that first hash every time you just like cash the round count a should hashes and then just read them later with that it's about 1.4 seconds one sorry okay all right sorry point eight
milliseconds for the original shuffle 800 and something milliseconds for the one that I wrote and the one that you wrote the one that the one that you wrote is the one that uses like you get per mmunity index for everything yeah it's that but it doesn't it's like slightly modified so it doesn't so that that first hashes I think stays the same across all shuffles so it just just calculates out that you know that first 90 hashes and then just reads them instead of doing them every single time oh I see so this you're just you're just talking about the round keys that are used to keep you
at the pivot right yep okay so the like the thing that's called hash bytes in at least like this optimized implementation like you're still not like you're still basically computing it yeah the pivot the pivot line or it could be a I'm just trying to find it's like okay no I guess there would be a hole five seven six if you want to see the four million validators which is the worst case it
takes one point two five seconds is that 40k awful million four million if so aside from shuffling there's also like recomputing validator balances do we have numbers on that we haven't done the tree education yet there's another question so this actually doesn't require this doesn't require the like fancy tree hash should I change one at a time thing that I talked about in the
last call this is just the validator balances because in an epoch transition you have to recompute basically everyone's balance right so this is just reconstructing the whole tree maybe we could create an issue we've also benchmark questions so that's all we come everyone can put run numbers in there it's a good idea I would I would just add one more thing to the performance issues and that is keeping state copies around because sometimes for example mmm-hmm the way the spec is written
right now to determine whether a block is valid basically you need to apply it and if it's not valid you need to roll back that basically means that you need to keep a couple of state copies around or at least somehow manage this complexity and there are a couple of obvious optimizations here to do which which are all centered around splitting up because apparently the spec is written with this beacon state in languages this big monolith but if you like one optimization that we will be doing is to split up that weakened state into more independent operations that maybe can be done in parallel but whatever but at least they can be applied partially and that would help a lot with these rewind all right so the
other yeah the other idea I've had for that is that basically you can store states going back exponential intervals so you can store the most recent state and store the state of log before and sort of say two blocks before it's or the state for blocks before and then if you get a fork that's like not a power of two then you would just recompute up to that point so that way you would just have to store like basically the size of the state multiplied by log n I think that's six maybe talking about like the fact that if I'm applying a block and I don't know if I need to revert it I'm as I'm doing that I need to store maybe in memory
more than just one state I'm having like 200 or some some diff on one plus something so like the effective state and memory that someone needs to handle is larger I guess like the mechanism I'm suggesting would definitely involve like holding log and copies of the state in memory where or at least a few copies of the state in memory like you can save to desk about some threshold and then you just like one of those copies would just be like Oh fourteen copy which is support I'm processing right now as you re computing like naively that means a lot of recomputing the Canadian yes the one
that is just worth like keeping in mind again is that you want an implementation that does not run out of memory in the worst case of 4 million validators and if he wants to have something that has like a very clearly balanced in memory of that like just having log n like copies of the state is one way to do it but then if you if you're using some journaling trick that or whatever then as long as it's like as long as you can bound memory consumption then they'll be fine you guys like as you do that exercise and splitting things up and seeing what can be parallelized please report back your findings yeah yeah totally cool okay anything else research
I want to get to talking about this wire protocol briefly so for that anything else on the research side okay Matt turned some of the notes we had about the wire API into a more concrete wire protocol some of you have already commented on that issue and I know there are there are a few more things that we need to respond and I'm gonna get to that today do we have any questions comments concerns that we want to address right now on the call relate to this I can answer feelings this question about RSA because I people ask me about
RSA so the lip PDP specification requires to keep everybody associated each node network default key pair is an RSA key pair I believe it's possible to change it but there does need to be some concept of a pair a network identifying key pair associated particular node on a limb p2p network because all over the pearbook and basically no identification logic in any the PDE library and the spec relies on that key pair so we can reuse key pairs from it just has to be some pair of some encryption algorithm no I understand no I just think depending on I'm a bit unsure in general with this with the
specification what what the sort of which level this is actually sitting at I mean is this supposed to be like a like complete specification of the entire protocol including you know basically which which which bins go where on the wire all the way up to you know which message types include which consensus objects the I added a little bit of a sort of additional context in there about the PP because I'm not sure what the sort of collective familiarity is with live PDP yeah it extends a little bit so so so the fundamental goal of the spec is to provide these are the
bikes that go over the wire these are the well the equivalent of endpoints that this those passages go to as well as like how they are transported compressed etc with an additional like bit of info about load P to be on top no I do think you know this general concept of like you know how know that entities work and how they handled everything doesn't really belong in the same spec is the sort of high-level message high-level message types the the being sent and stuff like that so and we also I would kind of recommend to maybe even sorry I'm talking so much maybe the other thing that could be useful would be to more clearly separate I mean there's already a separation between the
protocol that is spoken between two individual nodes that is probably gonna be used for syncing and the broadcast I feel like maybe it would be kind of nice to even make these like totally separate specifications so have these sort of like what goes on the broadcast channel be that once back and then having like the actual solar sync protocol as an independent specification I'm not sure if that will help but I feel it's kind of would be kind of useful I would mirror Fenix's comments and now with that it's a little bit sprawling as well and of encodings used like if you look there's like base64 base-16 there's newline the terminated strings there's
like binary little fields and then there is s SZ and just like lara i think i think we should take a look at how to simplify it a little bit as well in terms of encoding but you don't have to have so many of them just to read a message where which things are new lining code I think the prodigal lets you know the message ID just did the thing before the compression I don't have the word for it right now but sorry yeah but before the compression there is like the message identifier and that one is like it's flash separated string and
I assume it ends with a new line because yes so that's just an artifact so to support protocol you're right it's it's string the length of the accompanying string that ends up being abstracted away by the p2p libraries when you use them but they're there in food you sort of show the whole bytes that will go over the wire I do sorry this is Ronald I just want to make a point there because this spec was brought to my attention just a few minutes ago before joining the school so I wasn't able to
Wayne but but there are some misunderstandings as to how messages are carried particularly the fact that messages on each message is not prefixed with with a protocol itself you negotiate the protocol only once when you open the when you open each stream and that stream is then tagged with that protocol so some of these things will likely change or I'll make comments today yeah that's what I make so basically did the whole thing is basically this what protocols kind of like basically if you are assuming that the that all the messages are going to be carried across nipi two-piece multi stream likely the format is gonna be a bit more complex than what it is what it has there there's a front negotiation of
that and I think maybe this shouldn't actually be included in the spec because the spec the thing that we need to agree on maybe at this moment is more like a high-level message structure and then exactly how these things are going to be multiplex on single connection and how the whole negotiation and versioning and everything works I guess that can be left up to a into a later specification or to the live EVP multi-stream I think yeah I agree with you Felix I think ideally you'd be able to reference a well form well-written wealth backed out reference in the new b2b Docs which we're working on so hopefully we'll be there soon okay yeah we welcome your comments and we'll simplify and maybe
it's the meat of what is going on here and oh I did have a question regarding Frank had a question regarding Ian ours if nodes are discovered using in ours the information that is already available this means the client would be identified by an en our leaptv multi adder is used and it would be two vehicles the adders use and is not implementation detail and not a protocol requirement question so I guess the question is the where en ours versus multi adders come into play oh yeah I can I can help this so Frank isn't here I think he's listening on YouTube but
he's not in this call okay but basically so the idea is that yeah from from our side when it comes to the ear noise they're more like a discovery type of thing and I think his question was more related to how exactly yeah basically maybe what the specs should do is like document the assumptions that it has in terms of like pre obtained knowledge about the node that it's talking to so I think that would be enough in this specification to just say okay in order to talk order to speak this protocol with another node you need to know these five things about that node and then you know how these how this information is gonna be you know carried
or you know relate to you before you start speaking this particular wire protocol that can be left up to a difference back or you know I don't know and I feel that what kind of answer the question so that and when it comes to relationship between E&R and multi adder is basically that you can put multi other into in our so that kind of I guess that results the the discussion they are so if you do want to talk about nodes using first you can just put them Indian are and you gonna be fine okay thank you for the feedback let's we'll continue the discussion on this issue and maybe come up with a v2 or the 0.2 Lawrence can you I think we're getting
some feedback if you listen to YouTube okay cool well continue to go discussion here next week we'll come out with a like 0.2 draft addressing the comments and we'll go from there they all do there were some other questions about I'm not it is I hope this isn't the end of the meeting because Frank and I had some more things to say about this stuff so there's um the ongoing debate also seems to be about the serialization formats and this is actually pretty big chunk of this debate on the issue has been about the serialization format so we've kind of nice if we could you know settle this once and for all theorization is next up and any other
comments on the wire protocol for anyone okay next up our favorite topic so yeah so I feel like you know the from my point of view it looks like the the the same option seems to be to just say that you know consensus objects should ever it should be encoded using as a seed basically everywhere and then you know how the actual messages are encoded is a bit you know up to anyone's opinion I guess so it will be kind of nice to hear people's opinion like you know what what this is now specifically about the wire protocol that we speaking
between the nodes and the question is should it use Sze does it make anyone's life easier does it make everyone anyone strive harder options so from my point of view I can just say that basically we've been flying pretty well with ROP so far and the main advantages has over SSC is that it's an format that doesn't require a schema to decode so you can just it's a free-form type of thing just like Jason is so it means you can just you know could put a message there and then sort of like you don't actually have to know what type of object is
contained in a message in order to decode it and then with SC that's a bit different because it's basically just you know a collection of bytes that doesn't necessarily have any sort of self-describing structure at least as far as I know you understand right it does not have a certain structure what one thing was that especially regarding the SOS spec that I don't remember Kerala be put forward is that with SOS SSE with a little work you can have a specific byte offset so you don't have to read the and decodes wharfing to get
a specific field and that was one thing that is really missing from our P and the other thing is that Rao from Lee p2p said that they were looking forward to provide Wireshark detectors so that whatever we end up choosing as a wire saw realization format we have something robust to work with in Wireshark and overs network analyzer yeah so to that last comment why shark has been
you know some a desired feature for quite some time and it is definitely something that I wouldn't like to see but currently the nepeta be quarantine doesn't really have the capacity to to work on where shock detectors at the moment so I'll be happy to engage with anybody who needs some guidance to work on this and to maybe get a head start I have implemented watertight detectors in the past so definitely happy to talk about this and there are some ideas there are some things that need to happen in the b2b before we can facilitate creating motion detectors that can tap into the message and decrypt the message because there are as you know it b2b encrypt messages by
encrypt data by default so there is certain instrumentation that we need to provide need to expose secrets to to things like mahjongg detectors and other tools of the future so happy to talk about that offline so just click background on stuff I think the goal or the the decision made months ago was okay this works very well for consensus and at first look is fine on the wire and in case of simplicity to just go with until we decide that it is not not
sufficient for our needs to utilize it for both so that's just the context on the decision made there knowing that it was not necessarily an optimal wire decision but a simplistic something simple to just have across the board that's it for me I would generally feel that embedding FSB within our LP feel a bit weird cause they're so neat there Boyd so huh why is it why is it will specifically I'm
asking because basically any it's with our app is kind of easy I mean there's only really two things just listen listen bite erase and you could put anything in those batteries including is the blobs so I don't feel like it's a what do you mean from an implementation point of view or what do you mean yeah from an implementation point of view and also like because the value it add kind of minimal I mean in order to read the messages you pretty much need to know what's in them anyway good to make you know useful use of the data so having to custom serialization format where one of them doesn't really bar like what would it what would our be buy
exactly I'm not arguing for or LP in terms of you know I just want to provide like the basic fact is that you know you can decode it and you can see the high-level structure of the message without necessarily knowing which field is what and you this is something that is to my knowledge impossible with it but that doesn't mean that I'm like massively in favor of using LP for this to be honest maybe this is that could be kind of overkill actually it's it's funny let's say because in July when we are discussing about Sur ization format the main critic of our LP was that you didn't have any kind of schema and we
couldn't like know the offset without reading and becoming the wulfing so I guess it's either a plus or minus and we need to know for if to what what stance we take on that so the offset thing is kind of useful for I guess is most useful for like blocks and stuff because you do need to sometimes especially in inside of a consensus scripting environment like the EVM sometimes it's it can be desirable to get say I don't know like a bunch of health stations as an input and then be able to quickly find a specific one without necessarily decoding the
whole thing so and then you know for this type of thing having quite offsets is very useful but for p2p messages usually you would you'd have to decode the whole thing anyways because you're likely gonna deal with every part of it and being able to quickly access a tiny portion of the PDP message isn't something that has come up a lot in the past I would counter that on embedded devices and on upon any kind of resource restricted devices is actually pretty common that you want X is just a subset of the message and then it's in order to gain access to some limited sort of functionality maybe not the full thing
but to be able to run a specific part of or use a specific part of the information on limits it's actually a big asset to not have to go through too much decoding in order to get there it's not I I agree that there's a stronger case for it in inside box and so on but [Music] anytime you want partial information this is a useful property but I'll just add a trivial example here
like memory usage right if you have to decode stuff you have to usually put it in a separate memory location or you have to build up a complex structure that points out where the data is before you can use it so you will actually let me draw so in the needs of work yeah so you quickly just grow then the amount of stuff you need to know before you you can use the message so given that you know these these messages are carried on a on a byte stream I guess I mean in terms of the p2p it's likely gonna be carried on some content is P connection and you'll have to read from that TCP connection into a memory buffer and so
if you are a on a very very resource constrained device then it can be beneficial to actually walk the data as you are receiving it and then you know discarding the beginning of the data while you're already working on the next chunk or something like that and so these types of things we've actually done them with our LP in the past in the context of transaction signing on a hardware wallet so there was an implementation by Nick Johnson for sort of a for an oral processor that could essentially grab useful bits out of a transaction while receiving it so but in order to do that you actually need to be
able to decode any streaming fashion so we have a streaming a plantation of our LP available so that's like for us this is kind of a non-problem but I'm pretty sure that most implementations of our LP that you will find on the Internet are not streaming so they don't actually allow you to do this kind of thing but in general I mean for most implementations what they're likely gonna end up doing is just checking how long the messages and then receiving the whole thing and into one buffer and then working off of this buffer so working in a very resource constrained environment where you don't actually have enough memory to even fit one single Network message into memory all at once I guess yeah will
you'll have to come up with a workaround there why do you agree or is it yeah I can agree with that it was more mental a general comment that a very kind of additional encoding and and difference in in in format limits the design space of what kind of smart things you can do I mean I'll just leave it at that for now okay so I guess we won't really come to consensus today about you know this seems to be the core issue around these
two is series asian formats because we keep going back to it i I don't think we're gonna decide today i with respect to the discovery protocol 75 I know that our LP is currently baked into it is that decision I mean is that some sort of technical requirement in there or does it not really care what so as a should what is using so it is actually kind of a I mean we're coming I'm not sure are we gonna talk about this v5 in general now should I just I don't know where is it we are maybe more about the
serialization so I mean we've been using our LP for a long time for Ford for discovery purposes and I feel like it's been a bit beneficial there because for the tht the main problem has been that sometimes it is kind of necessary to extend the protocol in a in a backwards compatible way and for that it's actually really useful to have this freeform format because you can always just tag another field onto the message and know that previous versions of the kind of gonna ignore it and because they like ever since eep-eep was published which way more than a year ago implementations actually required to ignore
additional data in the message but it's still encoded in the same way as everything else which is you know as a field and some ROP list so it means that we can actually tunnel like for for newer clients that need a certain feature we can actually tunnel arbitrary data through the protocol if we need to and and this is something that I think is this is very very useful particularly for the DHT because well yeah implementations can be in various stages of compatibility with respect versions and stuff like that so it's it's kind of useful to be able to just you know hextech something into the protocol at a later point in time if it's needed and so this is actually the main reason I
mean from it from another point of view I mean it might as well use a sn1 or something we don't really care probably it should even I guess it's it's kind of useful to have it as a binary format but also the other thing is that for discovery specifically there's no requirement to have a to have a serialization format that is consensus safe as in that it doesn't need to be a single canonical encoding for everything because yeah well it's basically we just need any serialization format and yeah well we've been just been using our LP because that's also been used in a previous version and in the IANA aspect we've also used our OPA and there it's
actually required well it's not well doesn't have to be oral T specifically but the way this works is because you know it with collection of key value pairs you kind of need something that can represent that but I guess you could even build that with with as I see if you if you absolutely want you to write because you really need just like you have a variable length lists so okay thank you let's table serialization for the rest of the call and we are we're actually at an hour and a half but if you could give us an update on 35 and status and things moving there that'd be great before we close the call okay yeah so what we've been doing is basically we have now
published a a sort of draft of the specification over in the in the in the FPTP repository we're still working on it just now I am reverting the requirements document and I will add the topic discovery specification as well as the you know a spec in there probably next week so and then at that point we'll have a pretty much a complete specification but it's gonna have a few rough edges at the moment I'm also working on implementing the whole thing so I'm you know I don't know maybe like 40% done or something so I do have the basic structure in place and but I don't
have any any sort of code to share yet so that's that and yes specifically about that about that repo so this is something that Frank raised so for us it's been kind of hard to actually follow all the discussions because the PDP discussions have been spread around like so many different places so this is why Frank has been advocating on the meeting agenda - yeah well maybe move some of the PDP related spec work or something to the Deaf PDP report because here what would make our life easier but in general the way people seem to be constantly confused about the term deaf
p2p and seem to think that it's like a like I don't know like some you know I don't know it's kind of hard for me to understand why people are so confused about it but for us deaf PDP's just always been the name for like whatever a p2p system a theorem is using at the time and I will just continue seeing it that way so this is also why we've you know just started specifying the new stuff in that repo because for us this is like the repo for a 3mm p2p stuff so if you guys feel the same we can maybe consolidate the discussions there if you guys feel that you know another repo is needed or another place of discussion is needed or another place for specs is needed then we were happy to move our
stuff there I guess before us has just kind of been a bit weird because we feel like there is this repo and you know it's basically the main purpose of this report is to discuss and specify if mptp are things so I guess that's why this is why we are asking to maybe consolidate everything under if and FPTP because it's it's a place for that yeah that's fine with me we can maybe just make a tag related to - oh yeah there's also now I think there's now a feature on github that allows transferring issues so if you do want to transfer any issues
there or if anyone needs bright access to that repo we can we can give that out at the moment there's I think like three or four people who have write access to that repo but if anyone needs it because they want to prototype a specification or something we can we're totally happy to set it up cool before we close I have one question about one of the design decisions made on this this wire protocol inspect and it was airing on the side of more topics so like having topics for beacon block routes having topic for beacon block headers having talked the beacon block bodies rather
than having just a topic for the beacon chain having a topic for the shard chain number one etc of splitting it up into more granular topics one is there any sort of performance issue that you could see and maybe finding peers for these subtopics rather than just having the larger topics and - or there is this unexpected or a strange use of topics and discovery in general wait so are you talking about pub subtopics now award or discovery topics I guess both well yeah I guess they're distinct right because I could if I discover someone's interested
in shard one then I could use some subset some like extended set of chard one pub/sub targets so you're right there are distinct yeah so this is yeah this is basically the thing so I think like discovery topic structure is a different topic from the topic from the perfect gossip subtopic structure because because as long as you know you and and and all the other participants agree on like what what broad range of things to talk about you can just yeah like any at tagging the message with any topic is fine as long as it's gonna reach you so I think the main reason why we're even talking about having a sort
of topic based index in the in the peer discovery is because it's gonna make it easier for larger networks where maybe you're not going to be interested in everything every other node has to say but you know as as soon as you get connected to a certain good set of fuse that are roughly interested in the same things you you pretty sure to receive all the relevant broadcasts oh okay cool yeah I thank you for your answer and I kind of sorted through something else I asked the question okay cool heads up I think I mentioned this last call we are planning on doing a some sort of gathering before what very likely before
Eid Khan in Sydney there's enough of us there I think it would be useful potential topics and breakouts would be like a Azam slash state execution slash phase two stuff like client stuff and testing slash tests nets depending on where people are at we aren't certain of a date yet we're thinking maybe the middle day of the hackathon which is the ninth so that we don't detract from the opening and closing ceremonies by pulling a ton of us out of the heck phone but we still need to figure out location and details around that so I will keep y'all updated on that okay we're gonna close because
we're 10 minutes over thank you everyone a good call and talk to y'all soon oh and happy birthday thanks everyone thanks thank you [Music] [Music]
[Music] [Music]
[Music] [Music]
[Music] [Music] [Music] [Music]
[Music] [Music]
[Music] [Music]
[Music] [Music] [Music] [Music]
[Music] [Music]
[Music]