EIP-4444 Bound Historical Data (Breakout #15)

Transcript

work incrementally to to start thinking of what we need to do to make that a reality to make this initial proof of concept a reality and that's why i added those agenda items on github which is in particular you know what kind of format we should use which kind of client we should use and all that stuff and also we talked with some portal people that i think are in this call uh that could also give us an idea of how porzin could facilitate for forest and yeah that stuff i also just want to say really fast that

i started recording this call so if anybody's not comfortable with being on a recorded call i would go ahead and drop off now um yeah thanks for that info though george um we may want to wait like one more minute i think people are still still turning in here we go hello because piper said he was going to join and i'm not sure if anyone here is someone here else from the portal network uh because i do think we could have like a pretty helpful conversation there just to like uh at least

you know i think for me it'd be helpful to hear more about yeah how they see that fitting in here because i do definitely think that we want something like the pearl network to serve historic data and so it would support four fours quite nicely i'm i'm actually working on a portal network but it's lit p2p based so not directly with the portal group but maybe an alternate solution cool yeah i actually think someone showed me what you're working on on a separate call we are we're working on some of the like client stuff and talking about the portal network uh

and then yeah i think uh well separately yeah um cool good to see you i mean also i think uh regardless of portal network ipfs would be another place to store um historical data as one of the easy backends you know like torrent is the obvious one ipfs is another obvious one so that's another uh useful thing to have expertise in ipfs right which you are ipf expert right george yes since last night i am now an ipf software um okay speaking of do you want to walk

through the agenda items that you put down here george i think we can start moving ahead all right i mean look it's it's a rough agenda that i made up on the spot so i think like the way i i started thinking about this is that um we should think of how we expect nodes and clients in the future to use historical data like um okay let's see that we have a file that includes historical data perhaps per year per month we don't care at this point and we get this through torrent or portal network or whatever how do we want clients to consume this data so for example

the most obvious thing to do is have an import functionality um and and import that into like the local state of the client and have the client manage this another way that rai started thinking about is whether we want to have like you know be able to use the the the historical data archive files for for queries like the databases i'm not sure if that's worth having i think it might complicate things but if this is also something worth considering like how we expect the ux to be and then you know the next agenda item is given the ux that we want to support how do we expect the format to be like what

kind of encoding do we want um you guys are writing stuff but like i'm not monitoring it while i'm talking so if this is something relevant uh feel free to interrupt um i think yeah uh so what kind of encoding we want from the historical archives like do we want the binary encoding we want to be it's going to be s1 data so llp stuff do we want to use i know that some clients like best who have rlp export function how does that work i don't know so do we want to just use that we want to i'm not sure what we want to do here so that's the second

agenda item and the third is you know towards moving toward the proof of concept thing decide which client we want to target first and as someone mentioned you know in theory we just need one export functionality one client to export everything and we want all the clients to be able to import like we don't it's not necessary for all clients to be able to export since we just want one good export and everyone's gonna use that um so yeah that's the agenda and then you know have some discussion with the portal network group about what how that would look like and how integration would look like and that stuff does anyone think we should add something to the agenda or subtract

something i think that makes a lot of sense i think the only thing i would add is just like an initial question of what do we think the biggest blockers for for fores is in general um it feels like to me the biggest sorts of blockers are just going to be convincing people that this is a good thing to do i don't feel like any of the technical discussions that we're like looking to have are going to be things that we don't have answers to or we don't really have things that might solve those problems it's more just by shedding what the best way we think is to do it i don't know if anybody disagrees with that sentiment

generally no yeah i don't disagree so then the question is what can we do to convince people right and i think that's where then we like okay let's build prototypes let's get like well-known industry players on board all that sort of thing yeah okay that that makes sense to me mikhail um i'm this is related i'm thinking about the amount of like the amount of time or the the size of the period of time for which the history retains and retains under the guarantee that it's stored on the nodes in the network probably this one year is too much

if we want to increase the potential size of a block but potentially increase the size of a block i i've been doing some very rough calculations and they would have like five like what is the maximum uh size of what character is two megabytes right which is it really two megabytes this size of the block with current gas limit uh the box size of a block it's a kind of two megabytes right okay and if we have this like let me do it's going to be kind of a terabyte of

data for one year right if we have this kind of blocks of course we will not have if every block a size that big but yeah that's that's fair so yep um yeah i guess the question then is a little bit if we you know if we make a decision on this time period how difficult is going to be to to tune that over time is this sort of a decision where we're going to say you know here is the period in which you can expect we we're going to guarantee the clients will maintain that data and then

in six months or one year or something some time here we change that is that going to be a really difficult thing to do because i'm like a little bit worried about starting out with a release a lot smaller like three months because now that's definitely a different guarantee than some people might be used to since we're moving into this new world where you expect to have all historic historical data and now you don't have as much of it and going from all to three months is a little bit bigger of a jump than like all over one year by that you have the assumption that there are use cases that use historical data from three months to a year that

there are more of those use cases than use cases that use historical data from the entire history basically like recent history is more important than less recent history right yeah i mean i think i definitely think more recent is more important and it's not clear to me where it starts becoming like exponentially less important if it ever does but i would guess that there are still a fair number of people using things between a three and 12 month period and the less people we can upset with this change the better so i'm just trying to figure out like is this a decision that we're about to make that's going to be very difficult to change and tune or is this something that

is okay to tune i think it's okay to tune but also in terms of like de-risking this because it is such a huge change it makes sense to say yeah start with something very big where you almost kind of don't notice it and then once we have it working and we have all these other methods and people feel comfortable like going to torrent networks to get the history let's say things like this then we can say okay let's ratchet it down to like six months um and eventually there's gonna be you know there's gonna be some upper bound on like the consensus needs this much history to do anything so like you know it's gonna be a certain amount of time um that we can probably go slower than a year

if we need to what what are most people expecting the the weak subjectivity how like how far is the weak subjectivity checkpoint expected to trail whatever the head is so right now it's it's it's embarrassingly small it's like 14 days and we want this to get bigger uh but like i you know i don't think it will under the current model i don't think it will ever get bigger than like say eight months or something so it's true like one fundamental thing here is that you know the prune needs to be bigger than the week's objectivity period but um you know i don't think

that's going to be too big anyway as long as like it looks right now um um yeah i mean i think that yeah i think you know if we cut it down from one year to six months or something that would that would probably be okay i think one year is generally okay at this point in time it's just a matter of after the merge are we going to do something like 4488 that's to me where i think that the historical data will start going really quickly because it's just think it's

unrealistic for us to have consistent two megabyte blocks because that's assuming 30 million gas is used so realistically it's closer to like under one megabyte and then for the most part we're not pushing 50-60 percent of roll-up data yet so that's still a ways out where we're seeing that much uh block growth so if we do end up in that world where we're starting to have really cheap block data then we should seriously consider and so i guess it's a matter of do we think that four fours is not going to become useful until the end of next year in which case we should probably just start planning with the intention that

there is going to be a mechanism that reduces the call data cost significantly and we should set the um the pruning period to be relatively small hmm even trying to do 44.88 before the merge is possible i think like that's that's probably won't happen at this point okay i think tim said he talked to a lot of client devs and people want to focus on the merge and that's totally understandable i think we should focus okay so i don't think that i don't think 4488 will happen uh before the merge anymore which kind of gives us a little bit more flexibility with four fours we can just spend a little time focusing on that rather than trying to

you know combine them do them at the same time so just a bit of the mental model i have is like you know i don't know if it makes sense for this call to bike shed between one year two years six months like do this judging however like i think the mental model is that four fours should guarantee that clients can fetch data to do weak subjectivity sync over the p2p network but you know over time they will not be guaranteed to fetch any other data you know like potentially if we start shrinking the period uh they will get for a quick subjectivity thing but anything else they want they might not get and if clients start

working towards that assumption uh you know we can start with a big thing like one year but then we can start shrinking and more mechanisms will come that this will be easier for clients i don't know if this makes sense as a that seems fine to me so i think that we can like close that sort of question and talk about what bikeshare another time so i okay that kind of leads us to the question you asked though is how will historical data be consumed by clients and the way that i've sort of been understanding this is i think the ideal world is is that it's still built into clients it's just

you not using the dev pdp protocol for the most part and so it depends on exactly what the solution is maybe we don't if it's bittorrent we don't want to tie that into the client that should be something that we just download you know out of band but i think that if we are having a solution where it's just a list of mirrors that a lot of institutions or other people are providing that's something that very well could just live within the client and the client just attempts to download the blocks from those servers server lists so are you imagining though that a client then has all the code to process all of the history forever or as a second quote not necessarily yeah i'm not necessarily

um anticipating that i think that like a it's decision of the client that they want to have that code and then b i was thinking this is more in terms of the orchestration wrapper around potentially multiple releases just so that they don't you don't have to just go acquire those yourself and deal with the raw block data but i think like in the beginning it's unlikely that clients will start removing old code immediately and so having a simple way of allowing people to continue doing that is relatively helpful and i don't think it's a lot of overhead yeah that makes sense

is that something we feel like we should try and build out in terms of prototypes is this shim client that can process history since the beginning probably um yeah i think that the thing is i don't know when that that software would come into play like i don't know maybe marius can say when because it's not important until clients say we're not gonna support frontier anymore or we're not going to support pre 1559 blocks at that point then we will need additional software so if i don't know mario is that some of

you guys have discussed like if we do have an ability the ability to store prune historical data will you start removing those those old code paths he may not be available i don't know i mean i guess like you guys tell me how important you think it is like i think it is an important piece i don't think that it's a question of whether it can be done it's just engineering work to do and so to it to me it's like kind of important to determine is that something on client developers very soon road map after four fours like as soon as four fours is

uh you know is able to make the data available they say let's just start getting rid of stuff then i definitely think that should be a focus but if they're not anticipating on removing that functionality until 2023 then i think we could just focus more on this data format and then retrieval mechanisms so just on the original question to understand a bit more so let's say that i got the archive file from wherever like list of mirrors or whatever so i have a an archive file that contains i don't know blocks from the past four years and i import this into the client does the client need to like i don't know run all the evm and do everything or does it just like trust the blocks and provides them through the json rpc

api what does it have to do to import them like how does the ux and logic look like there i think you should get it onto oh sorry guys go ahead it should just be able to check it through hashing and we have would add one accumulator that commits to all the previous history okay so it doesn't run like all the transactions and it doesn't do the entire thing it just uh does some hashing to establish ground it depends a little bit on what artifacts your you know you want to be able to have if you're trying to build up an archive

node you would want to run all of those so you can generate all the intermediate states or the state diffs um i i guess like that could be part of the download but i think i guess that's like a question too is like what exactly are we promising to make available in my mind the bare minimum is just blocks like i mean that is the thing that we have to provide is all the block bodies and then potentially we might also make available all the receipts but then you start moving into this realm where it's like should we also make it possible to download any historical state and i think that that that would be where i would draw the line is that would not be something i would want to provide is providing historical state uh tri data

and so that would be things that that if they wanted that information those artifacts that generate themselves in the just to be clear the idea with having the receipts is that you can then verify the receipts route committed to in the block exactly yeah okay and the receipts are 80 gigabytes so it's more manageable than storing a lot of historical data i i don't have it like part of me would just say let's just serve block bodies because you can generate the receipts from the block bodies and that to me feels like it's a it's a little bit simpler less data anything that we can make this simpler

and less amount of data i think we can get more people on board with um to serve that data and i think that creating the largest pool of people serving that data is important guarantee yeah mary is just sudden chad blocks receipts and then snapshots every so often right so it could be like the state snapshot at the 1559 fork things like this um sounds like a good idea what's the use case for the snapshots generally it saves a bunch of work if you can trust the state right and then you do want to make an archive node but only from

4k forward right yeah that's fair so i think yes already have some import functionality does it can it can it already import blocked bodies and receipts are we like i think it's just of course block rlps so not receipts so it would just go ahead and generate that themselves i think uh all clients support um import sub command or most concept support import sub command that takes the block rlp body so is that like saying that the import

functionality is already done and we just like is this good enough for us for in terms of blockbuster design um i yeah i would need to look a little bit more and see the exact way it's it handles block bodies and how it handles a large amount like millions of block bodies i i think it's generally okay though like you give it magnet genesis and all the blocks i don't see why i wouldn't just go ahead and build the chain exactly like you provided it but the you know the question is do we want to do rlp and i think um i think we should look at using soc to be honest

so what about like block 20 block funny just like a random very old block like because uh well yeah either way it's like i don't think we can get rid of rlp on this front because like what would that look like i mean i guess you could define a separate ssd schema yeah and then it's fine uh but the thing is to like interpret that data like the the shim client or whatever we want to call it is gonna be you know like they'll be rlp route teams right so that's an open question i think so i think if we go to the ssc rep we won't actually end up getting rid of rlp

we'll end up with rop and ssc because you need rlp to be able to do certain types of verification and so the only compelling reason that i would say for using ssc would be to have the large scale merklization which allows you to have a single hash that identifies a huge blob of this stuff and let you get at individual components of it um and and i have no idea if that's compelling or not right like like it's um there's other ways to accomplish that um right but we don't have any ssc hashes in our protocol um for execution chain and so like we don't

have the right hashes to be able to verify ssc objects so everything would have to be converted into their rlp objects before you could actually verify them so i don't think that there's a big win there um at the like getting rid of rlp layer it the the only win i think that makes that compelling is something that just makes ssd add features but we don't actually get rid of rlp can i comment on this so i created for encoding we definitely stick with rlp just because it's used everywhere and difficult to remove at the same time however the commitment the part that mercurizes or hashes the data we can have both because we only end up

having one 32 byte object at the very end and then the schema like how do you commit over all the data there can be many of them and once we have these well-defined snapshots of historical data you can just compute each of these different versions of it the data is the same as long as there's some software that can reconstruct and recompute the commitment then it can be used for whatever use case you need that form of commitment for so and define the hashing schema as something that can be replaced

yeah i mean i think rlp makes sense where it already exists my question then would be are there like can we actually think of use cases today where we'd actually need commitments like this to the history uh because like what we're saying is there'll be some like year-long art archive of the chain and then we'll mercalize it and get some root is that helpful to anyone um can can anybody tie this together in a cohesive thing i i know that there's one thing being talked about which is making large uh you know the first million blocks the second million blocks right is that kind of what we're talking about here is taking a million blocks bundling up them up together as a single giant chunk with

in some format having a single hash that identifies all of that and and then what are we serving them from places are we making a torrent is this a new protocol yeah so i don't i don't think that that we've you know we're hoping that there's many different ways of accessing that data um the way i've just been thinking about it just in simple terms like to me that's a black box is how to access the data and we're discussing different ways portal network is one way that i think could be able to access data where you just request like i want these blocks and then you get the blocks the way that i think about it just in the simplest terms is that you know there would be a list of mirrors of servers that for providing all of the blocks they've just

elected that they would provide them they post on some smart contracts saying here's my server address i'll provide the box and then i can just go down that list and to request the blocks from that um you know the torrent is also like another potential option but i i'm thinking about it like in the most simplistic terms in that sense i don't i don't know about the bundling of blocks together this is something that i'm not sure about like is there a value of taking some number of blocks and then committing a single hash to all of those blocks because it feels like i am going to need to download them anyways and make sure that they are in a chain uh reduced number of requests that need to be sent is a benefit there right if

somebody is doing a million blocks than having to make a million requests versus a million divided by 20 or a million divided by 50 or whatever it is right i totally agree with that i think i'm more getting the the point where you were saying that there would be like a commitment or some sort of hash to authenticate those blocks against is that something we would want in addition to the authentication that already exists of the headers i think that type of commitment across a large scheme of blocks is only useful in the context of a client that is choosing to download the full history

marius your audio is cutting out pretty badly i don't know if it's better now so sorry there you go um uh i think since we have the header chain i would i would prefer having them accessible by header we can always uh sync the header chain and know exactly which header was uh at which block at which height and uh or which block was was where and we can just fill the header chain from from the torrent um and like either like yeah i i would i would prefer having it accessible by by by hash so we know the block hash and we we need the block data

for that and once we have the header chain uh we we know exactly which uh if the if the data that was just sent to us is correct and you can yeah like you can you can build optimizations on top of that uh as much as you want to by saying okay i want like i'm not i'm not i'm not i'm not sending a single header request but i'm sending like the state from from here to here and please give me like all the data and you can always check it against the header chain yep okay that makes sense um i guess like the last thing i wanted to say on the encoding topic

is yeah we have rlp and yeah it's you know very prevalent across the execution layer and it's going to be more work to use ssz but i think if we're trying to make this world where we do transition away from rlp at some point in the far future we should just stop building new things with rlp even if it's a little bit harder so if the goal here is to get rid of rlp then we have a deeper problem because if you want to verify the block body you need rlp because the transaction tree uses the x-ray patricia tree and the x-ray patricia tree uses rlp

so like rlp is still buried deep in there because the because in order to take a block body and verify it against the header you inherently need rlp to do that how about we keep rlp for just the existing types but then the date structures around it for example those that combine blocks or combine headers those should not be an rlp is that reasonable i don't think i quite understand yeah can you say that again so we have all these legacy types like transactions states block headers and so on in rlp we can keep those in rlp

but the all the additional types we defined for this eip could be in a different encoding like what kind of types are you thinking about for like the verification for example when we have um a series of block headers like a long set of block headers like a million block headers how do you encode that that group is one way that we could start moving to something else there so proto is the suggestion here that so maybe just as a straw man our data format is block headers chunked in

[Music] chunks of 1024 and so we take the first 1024 headers and we put them in an rlp list with a max size of 1024 with the with the the element type being right by lists or yeah byte lists um and so right the the data structure for the bundle of block headers is is rop inside of that it's it's rlp encoded sorry i said that backwards the the data type for the bundle of block headers is an ssc data type and inside of it are a bunch of opaque rop strings that represent the encoded block headers that will be a good example i guess

i'm still not sure that i understand the use case for bundles other than than for clients that want to re-execute the full chain um which is a totally valid use case i'm just not sure that we have i'm curious what the other use cases are which part what's the alternative to bundles um all of them like to me when we say bundle we're just saying like there's a million blocks and then uh yeah i guess i i don't understand i think paper means why are we chunking when i just everything at once like why are we saying there's chunks just minimize when does someone want like blocks

1 million to two million like what is the use case up there when does someone want block one million four hundred and seventy two thousand whatever in the past uh when you want a specific block it's because there's a transaction in it you wanna look at where there's a receipt in it that you want to look at that's that's the individual block use cases um authenticate that header so so the authentication can be done with an accumulator that doesn't have to contain all of the data and that's something that we can like i guess that is something that we haven't considered much is these like on-demand requests which i think is very

different from how you think mostly about portal network um i think a lot of the discussions we've had so far have focused on you're going to just download all the blocks because you want to have some sort of client with a header chain running is do you think that that's something that we should start considering more is it if somebody is trying to request this historical block they want a certain transaction in it we should have facilities for them to do that or is that out of scope for four fours and that's something they should look at portal network for well i don't know what people intend to use this for so what i would intend to use this for would be for on-demand retrieval of individual data because the reason that

clients are ditching this data is because they don't want to hold on to it right because it takes up a ton of disk so if a client wants to fetch an old transaction from an old block and everything's bundled up then it's got to download 10 000 blocks to get it one transaction that's not great um we please separate authentication of data or like the commitment to the data separate from the encoding because if a client asks for one piece of data we can have a different encoding for the response than for the encoding of the local storage of like the full snapshot

or a snapshot divided in chunks or whatever it may be there are two different things here and [Music] the authentication of data you can have many different commitment schemes and you can optimize for the user that request the data and that can be different than the optimization for the rotation of the the snapshot or like the how we collect the fill data somewhere yeah it sounds a bit to me like we're mixing this idea of like the history from like yeah i went like one block industry

for me so far four fours is about we're passing around like chunks of a year of history at a time and it's not so much like the question of how do i get one block from three years ago is different than how do i get the snapshot from three years ago and uh we could like we could have a conversation about where that boundary is i think the less scope we have in far forwards the better for the specific eip yeah so if you want to constrain the scope then individual retrieval is a good thing to wipe off the table and bundled these up in big blobs that are of some

you know arbitrary but reasonable size and all that we're focused on is archive node style re-execution of the full history i would say the commitment to the full data can still also serve individual requests like it could be a efficient way to exactly figure out that this one smaller piece is correct that it can be authenticated with some miracle proof type of thing with the larger commitment or for all of the data but we should focus on getting

the imports right because we're trying to remove all of this data from the default node so i think that if portal network was finished right now then i don't think we have a need for like this other thing that we're talking about right now for four fours the issue is of course portal network is a much harder problem than here's a torrent with you know the last seven years of history like a tournament seven years of history is something that we could throw together you know in weeks portal network is something that will take months years quarters to build and so i i think that maybe the question i have for piper is how far away is portal

network from being able to serve all of history um yeah i have a decent answer for that um we did roadmap planning last week and looked at what's coming we've shifted priority to history network over from state because of four fours and because of the you know immediate demand i think that it is reasonable to expect us with no additional help to have a version of the history network that serves all headers all block bodies and all receipts from history with some level of reliability by the end of 20 but by the end of next year

and with a like functional prototypes and proof of concept stuff showing up in the six to six to eight month range into 2022 so and that has the ability to be accelerated with additional people um getting in on this um but but but it's probably not gonna it's not gonna be two or three months right we're still um looking at q two q three q four next year my gut given those timelines those times are about what i expected to hear um my gut is is for four fours in order to get it out asap

um we need something like anything literally we need some way for people to be able to download the history there are very easy solutions to this like just one big torrent file that has everything up until january 2022 or something you know and that solves like 90 of this problem or 99 of this problem for a year at least because if the expiry happens after a year that buys us one year by just having and if portal network's done by the end of that year then we don't need to do anything else like we can just have one single torrent file that you know is a solved problem that buys us a year for full network finish and if portal network's not done in a year you just do another torrent file like do a second one um i i'm hesitant to spend like a huge

amount of time engineering some really complex solution to to downloading history that's just going to be replaced by the portal network in 12 months like that that's my biggest worry is that you know we are building two things one of which we need right now and one of which is much every i think every i think everybody agrees someone correct me here that is the best long-term solution which is the portal network because it's fully decentralized you know you can pull individual blocks you don't need to get all of history at once uh works for both archive nodes and it works for like clients like it solves everything at once and so i think long term we want for network short term we need something so we don't have to wait for a year for 444.

so is there a world in which we have both because it kind of seems like to me they're almost addressing different issues right like there's just very separately for forza is saying here's the history how can we pass it around to whoever wants it and almost a layer on top is the pearl network right because you still have a question like how do i bootstrap a portal network node so so i think that this relay yeah so i think once portal network's done you boost our world network node by just connecting to the portable gossip network and just start downloading it's like a light client may say i only want block four million three hundred and twenty two thousand whereas an archive node may say hey portal network start giving me all the blocks from one and start counting up and i believe portal

network can solve both of those quite well um the caveat is is i do suspect that no matter how good the network is a single giant bittorrent file is going to be faster if you do in fact want to sync eight years of data a bittorrent is i don't see any any solution that's going to be out bitcoin for speed that being said again from an engineering standpoint it's expensive to build a solution and if portal network you know does good enough and we have all of history in a bittorrent up to january 2021 to bootstrap from um i'm just i'm hesitant to synchronize your engineering effort into two solutions that overlap like 80 or 90 percent

can i suggest a straw man option here because micah i i agree with you i i i think everything you said there is at least my sentiment is mirrored there um so a simple solution here is either go real simple and don't invent anything new here and we bootstrap a couple of torrents and and we're done it's like quite easy we have to you know come up with the you know some some way for everybody to agree that yes these are the torrents and yes we're we're happy to uh uh you know hard code all of these uh uh what are we talking about magnet

links or whatever the hashes are that right commit to to the data and the torrents and that's super easy and straightforward and boring and and and does a good job of doing what it's supposed to do um [Music] and in theory does still allow all of the other stuff that we want which we talked about with ssc data structures being able to index into it and grab specific ones turns out you can do that with torrents too if you know things are split up across a couple of files and things like that and maybe the files within the torrent are blocks of you know a thousand things at a time encoded with ssc and all of that seems like pretty uncontroversial and straightforward to

implement um and then clients need to support ingesting this stuff which means embedding a torrent client and being able to process the data files so the work here like we could bike shed over the format and stuff but the work here is somebody to sit down and just go ahead and write that down and bang out a proof of concept for it right like um i guess what am i missing like where's the point of contention there in that in that proposal a torrent full of big files that are ssc encoded um full of you know whatever data it is that we're betting

in here and they're available for download and we have a bunch of people who see them i don't i don't i don't think you're missing anything i think that's aligned with at least how like george and matt and i have been thinking about it uh the question i think i think the conversation was having was like how does that align with yeah something like the portal network because then we have this question of just a second ago of like are we somehow duplicating engineering efforts like uh if we're on board with like moving forward with this like torrent idea for like the year to two year mark and then you know once the portal never comes online we can reevaluate that's fine

um do we consider it wasted work if we do build out this like quote torrent client everywhere or at least in some clients i don't think that the clients even need to embed the torrent client i think that you download the files using a normal torrent client and then you point your client at the file when they're done downloading so does this so yeah marius basically said uh you might not like the torrents do you want to chime in marius if you can yeah so um because i think what piper said generally makes sense to me but yes like i don't like i don't see us dropping history

uh without any like really trustless and and decentralized and reliable way to get the data and uh like i don't see torrents as really reliable and i also don't see as like like getting blocks out of protocol as a reliable way to to to like to really know that this history will always be there what are you saying what are you considering let's drop sorry against my ass is really quiet are you saying you're not going to drop historical data that clients already have or you're not going to implement four fours i.e stop serving yeah i think like

you will continue serving until there's a decentralized way of i i i believe so like i i that's of course i cannot i cannot make the decision myself but uh from from my point of view uh i would not drop uh drop the requirements for the equality mark maria i would push on that quite hard because torrents are decentralized peer-to-peer infrastructure that are no more or less robust i'm not disputing i'm not disputing the decentralized uh fashion of of torrents i'm disputing the reliable the reliability i think people so

because i want to focus that reliability piece i don't think holds up either because right now that's already starting to show up in the existing peer-to-peer network we've got clients starting to drop historical receipts and not serve them so so this is already a thing that's showing up and a torrent that has one reliable peer seating it is still available and this is something that we can absolutely sick the community on and that people will just perpetually host it for a long long time very easily so so i want to i just want to challenge that idea that it's not reliable because

we can definitely make the torrent solution reliably available for the for for the next 10 years without problem well i think we can i think i think we specifically can argue that torrents are more reliable than the current ethereum gossip network and the reason for that is because it's much easier for a person who just wants to help out and they have much a disk to serve a torrent than it is to run a full client and so it basically allows us to separate out the hey i've got a bunch of disks but i don't really want to run a full ethereum client i'm going to just serve you know this data and you can serve that off of a hard drive a spinning disk for example you don't need an ssd or an nvme drive

you can just have some old spinning disc you had from five years ago put the torrent on it and and seed it to the world and so i think that my argument is is that well i can see the argument for torrents aren't reliable i would argue that they are more reliable than the current ethereum gossip network for storing for serving an archive state what this incentive would be for anyone to altruistically yeah really um altruistically just serve this kind of blobs we have evidence that there are at least 9 000 plus altruistic nodes existing on our current peer-to-peer network that

are serving this data right now who are running nodes for whatever reason right like those are not all miners this is the cost of running a node currently so you have to do this but otherwise when it's just i'm running five different ethereum nodes with no actual like personal benefit at all so i think the i i will agree with mikael here a little bit um in that guest specifically i don't believe has an option for not uh downloading and keeping old state the other other clients like another mine for example just has a command line flag where you can just say no i'm not going to store it all and a lot of people who are

another mine do not serve the data and so i think there is an argument to be made that it's not altruism so much as laziness like it's if you have some need or reason to run an ethereum client and for whatever reason you choose guess because everybody chooses geth by default unless you do a bunch of work you're going to serve all the old data whereas if we switch over to a torrent those people who are like oh i want to run an ethereum node but i'm not going to go out of my way to be an altruist those people will stop serving and so we will reduce from lazy plus altruists to just altruists with the torrent option yeah and this like this is why i like the the idea of having a new protocol

uh a new networking protocol that that achieves this so basically like retire the the history history retrieval from from the current eth protocol implement eth 68 or 70 or whatever and create a new networking protocol that is only there to serve these these these old states and then we can build nodes that that just that just serve these these old states we can build our own torrent network and we can have even uh that's a node and implement both uh both uh networking protocols that seems definitively worse than a

torrent because if nodes have the option to be part of or not part of that network then it's the same concept as opting in or opting out of a torrent nodes are wanting to get rid of this data so being part of that history network means keeping the data and serving it and so now you're relying on not just anybody who wants to host a torrent and serve it but only people who want to host an ethereum node which with all of the complexity and heaviness that that involves plus the the history data so that seems like a like like like uh on this at least seems like

it's worse than the torrent solution in terms of reliability like like and then and then for example the portal network could tap into the same protocol and serve the same uh uh the same blocks on on on this protocol and we could have we're not going to do that though because we're not going to get on devp the portal network is explicitly like on its own separate like networking protocol on top of disk v5 for for specific reasons so yeah but if you want to build a separate network for storing and serving all the history data we're doing that it's called the portal network so if we want

to solve this with that then let's do it that way because we're already like way far down this path to build a decentralized network that serves the history but if we want a simple solution that serves big blocks of the history let's use boring solutions that are already well known and understood and and do it that way yeah and like i i agree like like i agree about that but i'm just saying that i'm not comfortable dropping our requirements before we have this decentralized solution like i i agree that we can build it and we can we can use a

the the uh we can do the bittorrent uh thingy or whatever torrent uh we want to do but i'm not comfortable uh matching the pull request to drop this historical states uh before we have uh some some some uh our own version of it okay so is there somebody from the geth team who wants to start building out a portal client in the guth code base because that's the solution for that right like um i i'm being a little like pushing here i get it but like like if if the only way that that i know that you're not

speaking for the geth team but if the only way that like you're comfortable you know accepting four fours and dropping this data out of the client is to is to have a an owned decentralized peer-to-peer solution for this that looks something like the portal network or is you know on one of our peer-to-peer networks and you want four force to happen then that the the answer to that is prioritizing work on on actually building that now yeah right now we're prior we're prioritizing work on the merge so uh like we have our hands full with this but yeah i think i think that um uh schultz is uh interested in in in uh

uh in doing something like this i'm like i'm not sure if like we haven't really discussed uh for forex for force uh in in uh in in the team yet um um because we have been focusing on in my opinion more important topics yeah but yeah i i i agree that i i agree that uh uh like uh uh gath building a a portal client at least on the on the on the history retrieval on the history network would be uh would be very important okay so i think we're about to hit the time that we had scheduled uh

so i think the takeaway here is that we need to probably better understand uh you know maybe do some more legwork to understand like what's an acceptable solution on what timelines um does that sound good i posted this thing in the chat where this eip tries to solve three different problems i'd love those problems to just be solved separately so first we have import functionality then we have distribution of data and then we have the final removal of the from the defaults where we don't serve historical state anymore

i think three is like the final goal of four fours but i think we should have one in advance and i think two is it's getting a little bit heated and to keep moving at least start with one i think four four four four only specifies three at the moment i don't know what eat that is we have one that's called four four four four four four four hp4444 is the i'm just being difficult for forest is the correct pronunciation it is not former force anyone who says four forces is wrong in the media what i got from the last point where we want to remove

the historical data surfing is that we cannot warrant this removal we cannot trust it until we have solved one and two yes i agree i agree with that i do think that we knew didn't need to solve one and two um i just want to be clear that the eip itself is only solving three it's just one can argue that it will never pass all core devs without one and two being solved separately already like we need to we need someone to write two more eips basically yes let's try and solve one and two and i think one is really simple two depends on either portal network or a torrent client or something else

but we should look at solving these things and then we can clear up for for us to move uh on yeah yep okay sounds good guys all right yep thanks everybody for coming i'll post the recording for this on the github issue and then discord and you know we'll just keep the conversation going um yeah we'll try and uh synthesize what we talked about a little bit i think we had some good discussion about how to solve one and so maybe we'll try and write something more concrete up about that

sounds good yep sounds good all right thanks everybody yeah thanks everyone talk to you soon yep thanks for joining bye everyone bye