Near Protocol: 'Blockchains Cannot Scale Without Sharding!' - Illia Polosukhin. Alex Skidanov 546

Transcript

you need to actually store all that state somewhere and for a billion users it's actually going to be a lot of State this usually everybody you know talks about GPS but actually one of the bigger bottlenecks right now across the space is State and so obviously no single computer no single kind of node can process all that can validate the whole network now you can rotate this validators validate what chart all the time every block they can be actually you know you can randomly select set of validators and they are able to validate this because they don't need to sink into The Shard they don't need to process every single transaction that ever hit that Shard they only need to look at this specific block the data

availability and consensus are merged together into a single single mechanic the essence of it is you you want to have a design that is not dependent on the underlying Hardware itself improving at a certain rate to be able to service user demands you want to have mechanisms beyond that type of scaling this episode is brought to you by nosis nosis builds decentralized infrastructure for the ethereum ecosystem with a rich history dating back to 2015 and products like safe cow swap or nosis chain nosis combines needs driven development with deep technical

expertise this year marks the launch of nosis pay the world's first decentralized payment network with agnosis card you can spend self- custody crypto at any Visa accepting Merchant around the world if you're an individual looking to live more onchain or a business looking to wh label the stack visit nois pay.com there are lots of ways you can join the nosis journey drop in the nosis Dow governance form become a nois validator with a single gno token and lowcost Hardware or deploy your product on the evm compatible and highly decentralized nosis chain get started today at noses.

iio cars one is one of the biggest node operators globally and help you stake your tokens on 45 plus networks like ethereum Cosmos Celestia and dydx more than 100,000 delegators stake with chus one including institutions like bitgo and Ledger sticking with corus one not only gets you the highest year but also the most robust security practices and infrastructure that are usually exclusive for institutions you can stake directly to cors one's public note from your wallet set up a white label note or use the recently launched product Opus to stake up to 8,000 e in a single transaction you can even offer

high yield staking to your own customers using their API your assets always remain in your custody so you can have complete peace of mind start taking today at chorus. one hello everyone welcome to epicenter today I have an amazing episode lined up for you we're talking to Alex and Ilia the co-founders of near which is a in production sharded blockchain with a lot of value writing on it uh specifically we will cover how near sharding actually works try to build it concept by concept into an integrated hole and then understand where they are in their journey to implement sharding Alex Andia

welcome to epicenter so at this point we've kind of done a few episodes with both of you uh maybe you could give you know uh short introduction to your to your backgrounds hi yeah I think at this point my background is all near everything that was before is irrelevant but uh yeah I I was working on a sharded database called msql before near for five years which is right now it's an in production sharded database and uh yeah before that I was at Microsoft yeah I mean my background is actually in machine learning AI uh I was at Google research prior to our startup adventures and uh I was uh one of the C

I was on the paper that introduced Transformers which is technology powering chpt M joury and other advancements and then with Alex we actually started origin near as AI company and uh realized we need fast cheap easy to use easy to build on blockchain because we wanted to uh use it ourselves for uh data crowdsourcing and some other data use cases for our company and end up pivoting to that in 2018 and uh yeah focusing on that ever since yeah let's get into sharding um blockchain

scalability so what is sharding overall and why has it been a generally difficult Target industrywide I think I mean maybe bro broad question is if if we imagine having a billion users coming in and using lochain as a me ofd as a Meo tracking ownership as a way to kind of coordinate resources and efforts uh you imagine that you need to have kind of a few things happening right one is you need to actually store all that state somewhere and for a billion users it's actually going to be a lot of state and this is usually everybody you know talks about GPS but

actually one of the bigger bottlenecks right now across the space is State uh and kind of its growth and so so that's probably problem number one problem number two is I mean you have billion users users try they transacting there's you know hundreds of thousands of applications obviously there's a lot of uh kind of transactions flying around and and a lot of processing that needs to happen and kind of people want to put more and more complex smart contracts and complex logic into this so you need to have throughput bandw and processing power to do this and so obviously no single computer no single kind of node can process all that can validate the whole network and so the way to support this

is one way or the other partition this computation partition this storage and partition this kind of band list to receive this and so people have kind of historically been talking about charting because that's how the two companies doing this right so Alex mention you know doing this kind of INB 2 world M SQL single store now is used by for 500 companies you know Google and Facebook have their own Solutions and it kind of seemed reasonable that this is an approach that you should take in blockchain now there are a lot of kind of problems that arise when you actually move into a permissionless

setup uh kind of compared to a permission setup that usually we two companies deal with I I also want to add Ilia said it's obvious that you need to share part uh processing I don't think it was obvious to everybody until recently so there were multiple blockchains which were uh like the favorite thing they they like to say was like look look at Visa look at how many transactions Visa processes and that's a world scale obviously we can do it on a single computer right but as a user you don't use Visa frequently right you use it like three times a day on a good day right and so finally now generally when near launched we makes multiple bets which were not obvious to everybody you know like when near from

day one we had named accounts where you can rotate Keys uh and and we had Charing and it wasn't obvious to everybody that it's useful and now suddenly very scalable blockchains which are not sharded get uh you know uh congested and they have no way of getting any more uh performant right and and similarly like you know when it comes to account abstraction like ethereum right now is uh switching to to that so so now finally it is becoming obvious to everybody that those were correct decisions yeah I mean to give an example right like any any kind of a single I call it single node blockchain right

which is something that every single node in the network needs to process every single transaction and store all the state right what it means is as soon as like let's say that network has a high capacity they also have a huge State growth they have kind of limit on how many trans action they can process because of the bandwidth and execution and so at some point there will be more demand like if if and this a very natural thing right the price for the transactions usually based on Supply demand and so while transactions are cheap they you know at some point there'll be more demand because they're so cheap to even just Spam spam it to try to get some Financial U benefit from

because the transaction fees are so cheap and so when that happens right you don't have a way to expand capacity so your your prices start to grow for everyone right and so so this leads to Now you kind of pricing out people who were using this blockchain originally for normal use cases because of the spam and kind of people trying to run Arbitrage for some other application and so that's kind of just the principle where like any single like kind of state machine right single estate machine will get overr and start Rising Fe right so kind of Solana is the you know contrasting example of course even ethereum and Bitcoin are based on the

idea that the miners or the validators and even the full nodes of these system have to process every transaction uh that is happening in the system Solana has taken that idea and said yes um we have a bunch of validators I think 1,800 or something like that currently and every transaction that goes so the Solana system has to be processed by every one of these validators they are assuming that okay these validators can be placed in data centers where um networking bandwidth is very high which means a lot they can ingest a lot of transactions from the network uh at a very high rate they also assume that the machines are very

performant so the work of accounting can be um you can assume that each machine can handle lots of uh transactions uh do do their do their accounting work and then Sol would also assume that okay maybe the history doesn't need to be stored by these machines they only need to store the currently what are the different accounts and what what balances they own and what they they kind of like a project like Solana assumes is the Improvement in in compute in terms of like bandwidth processing power and every resource is it kind of doubles on some time scale so some resources

double on 12 month time scale another one D might double on threee time scale but because of this doubling the capacity of the blockchain would keep growing at a certain rate and the hope is that the user growth is actually slower than the doubling rate of the underlying hardware and therefore you continue to have like a cheap blockchain whereas in the near case like near is the opposite approach where we where you say user growth can be way faster than the Improvement in Hardware so fundamentally you need to move away from the Paradigm of every validator or every node needing to store

first of all what are the balances of every account in the system are and then you also need to so basically no single machine may have a complete view on what the balances of every account on the near system is and then uh it might also be the case that there's some machine and there's some transaction on the network and that transaction happened but this machine is part of is is is part of part of processing that network but it never actually executed the transaction itself and so the essence of it is you you want to have a design that is not

dependent on the underlying Hardware itself improving at a certain rate to be able to service user demands you want to have mechanisms beyond that type of scaling yeah I would add that as I said it's not even about users it's it's actually in in a way it's sadly the economics the economics all this blockchains is such that you need to have kind of a way to expand capacity otherwise if you want to maintain low fees because it's at at some like at some level there will be a set like

saturation where some subset of people are willing to pay higher fees because they need planning like they try to capture some economic value from an exchange token launch whatever Trading and that in turn in increases price for everybody else right and so so salana and some other like kind of high capacity networks even though the idea that like hey we we have to enough capacity and it will grow over time the reality what happens is since that it gets slotted by transactions that are all trying to hit uh kind of the same Economic Opportunity and extracted value and so that people are willing to pay way higher fees than uh folks that are

uh potentially using it for you know other use cases right for payments for example and others and so that's kind of the the point is and this is to add to the fact that like yeah the state grows and everything requires vales to continuously expand their Hardware even uh just to continue maintaining the network so so I think like to me actually like past like three four months have been a really great validation and then this is not just about Sal base and other like kind of even you know bases centralized sequencer right it's a single server effectively but even that cannot CCH CH up with all the transaction that they need to process and so that's kind of an example of just like as soon as you have

enough economic activity you starting to get kind of this flot of transactions trying to capture that and you don't have any way to either isolate it and add that extra capacity for everybody else to to do this right and so example I like to use is Imagine Netflix you go to netf and first of all to like in the CD ecosystem it would ask you to choose which data center you want to watch from right do you want their arbitral data center or optimism data center uh a base or you know blast and then when you go there it says like actually first of all you need to bring your money from the other data center where you have your money uh if you want to pay for this movie or pay for watching the movie here and then second one is actually you know

because somebody else watches a a very popular movie you cannot watch this movie right now at the lower price you need to pay more so that's kind of the current state right is and what we want to do is you know you go you can pick any movie and you watch it and you pay kind of fixed fee right that like is predictable for everyone and so to do that right similarly how Netflix need to use Amazon that kind of scales under the hood right and is able to build more data centers kind of ahead of the demand that thatx has kind of similarly you need a network that is able to scale with demand and kind of you know in a way you know you have the supply demand curves and so like you want to Flat the supply curve

such that even as demand grows you kind of can maintain the their fixed fees right so this is the distinction between burst capacity and like average capacity in a sense where like a a system might only be using like X capacity certain times in a year but then suddenly like one application or the whole system might require 5x or 10x and that burst might happen very quickly and what you're saying is essentially that if the if the scalability properties of a system are only dependent on the underlying machines that the validators use then

that cannot change very quickly to adjust to burst demand like suddenly l of demand comes in machines can't be changed across the whole network that fast so there needs to be like some other mechanism where a burst happens and the system is also able to somehow respond and being a and be able to scale uh dynamically on a on a shorter shorter time Horizon and this is this is a property nearly every blockchain kind of like lacks today which is why you have gas congestion yeah and so specifically in last months we went from four shards to six shards increasing our capacity by 50% because

we had a couple of applications who had like massive growth we have hot which grew from zero to five million users it's like over mon million daily actives wasin the months and so we started to having this actually congestion in south of shards because of that and so instead of just okay everybody's now paying higher fees near Network out more capacity so yeah let's get into what you know what what are the different challenges with with build it building a sharded sharded blockchain so there's a couple of them uh first of all um there are certain uh changes to the user experience because

uh since nobody maintains the state of all the accounts if the transaction has to tach multiple accounts uh something needs to be done about it right and it's it's a it's a very large uh design space and uh uh generally we refer to those transactions a cross sh transactions and uh uh that's uh that's one big challenge uh the Second Challenge is uh in the network in which every node processes every transaction if you if you have a node uh and you're at a particular State you have very high certainty uh that every transaction was processed properly because you literally process each and every one of them right in the worst

case you can be in a situation where you're looking at a network which is not canonical right so maybe some other part of the network believes that the another set of transactions happened not this one right but that's a very different problem from someone literally you know like making something that doesn't make any sense according to the rules of the chain uh and so in the charlot chain because every node only applies a subset of transactions you need some checks and balances uh which ensure that that what you see is actually a correct state that results from the properly executed transactions uh and when you start digging deep into that problem into the problem of ensuring that everything is executed correctly uh then you you start

facing another problem where in order for like for almost any mechanic you can come up with that ensures that everything is executed correctly maybe with an exception of ZK proofs like once we start digging uh today we will we will see uh you need to be able to access certain information in order to perform the validation and that information could be made unavailable by malicious actors and so you need to have a sort of native you need to have a native mechanic on the blockchain which ensures that certain pieces of data are available to certain participants and cannot be concealed right and so those two challenges are

like one of the biggest challenges that exist there there are some others one interesting one would be that because the state is so big you need to have a way for people to either synchronize State very quickly or work without synchronizing state so I I think those four would be the most interesting ones right so like yeah essentially like imagine yourself like being an accountant of the near blockchain it's um it's a massive data structure you only have a small part of it the transaction comes so the first problem is okay you might not be able to process the transaction fully yourself because you

only have a part of that entire data structure so you can maybe make some changes to that part but then the transaction might hit okay now it needs to do a change on some other part that you don't have and that's a completely different accountant so you need to process you know some only a a part of the transaction and then it needs to be like that that relay beton needs to be kind of handed over to some other party and then they do that and and so on so like that's that's one that's one issue second issue is if you're only handling a part of that data structure which has contains

all the account balances you received that data from some place and then how do you even know that that data is genuine right so that's that's the problem of uh stateless validation or like how do I know that this data I'm receiving it's actually processed correctly in the past then kind of like okay if there's any mechanic to know that uh any kind of certificate that would tell me that this was Pro processed correctly in the past then the generators of that certificate kind of need to have had access to that data

in the first place but if that data wasn't there to the generators of like these certificates then um then they won't be able to generate certificates or certain data needs to be kept in a state where you can reach it in order to do something with it yeah so so couple comments so the first one what you you mentioned that you need to process part of it and then send it to someone else so it's also important that that message you're sending is not getting lost right um it has to be delivered with certificates you mentioned that you need to be to have certain data to create certificate it's also in many situations

the case that you need certain data to be able to verify the certificate actually like in near in in one sense like there's a lot of complexity because fundamentally like kind of like the accountants in your system do not have may not have access to the full data and so that leads to a lot of complexity but in a different sense near is um near is similar to other blockchains in it that it's just one data structure containing accounts and like smart contracts and their data and this is exactly the same as kind of like Bitcoin and ethereum where it's you're dealing

in the end with a single data structure near is managing the data structure in a different way but ultimately it's it's a single data structure like that's that's a unifying property across all of these systems is that right yeah so yeah you can think of near as like very large mapping from account IDs to State uh a big difference from other blockchains is that the account ID is not your key account ID think of it as like a domain name right so let you know I would be Alex near uh and it's not just a convenience it's not just something that is easier to convey to other people it's also um like if if you think about it if if the key is what identifies your

account then if for any reason you have reason to believe that the key is compromised your account is gone right you have to create a new one you have to try to move the assets not all the assets are movable right like you can imagine an nft which is not movable by design an nft you know like the first user of this feature it's not a movable nft that nft is gone forever right so near if if I have a reason to believe that my key is compromised I just change it right it also allows you to to do to like uh auction your account like like your account has a particular state that you think is of value you can literally sell your account there are services on here that allow doing that right and then this massive state of mapping but

but at the end of the day it's still a mapping from an account to State similar to bitcoin ethereum salana any other blockchain but that state is shed just just to be clear not Bitcoin because you txos but yes ethereum so that every account based blockchain but yeah you know like EXO you can think of it is has even less uh persistent account than than just a key uh and so this state of all the accounts it's split into multiple subsets right into multiple sets so uh today we split by by name right so there would be a contiguous set of accounts uh that lives on Shard one and there is a contiguous set of accounts that lives on Shard to uh and that those boundaries are not

they not immutable through the through the life of the blockchain and as a matter of fact they did change multiple times from near launch it was a single set near launch with the Single Shard right and then it was split into four and then in the recent months it was split two of them were split multi twice into two again so it's a and in the future that will be dynamic in the future the system will be changing the the boundaries as the you know like as the load changes automatically so I mean maybe one way to imagine it is like um in your like in the postal system in your city most likely like your city is kind of like divided into these different

regions each with a different postal code right where I live they're called yeah plzs or something like that and so each kind of like region of the city will have will have a number and the city will be partitioned into various different postal codes and you'll have like a post office in every every code essentially and can imagine like near it's it's taking that data structure you can think of that as the city and then breaking it down into like these these areas these shards and the dynamism is like if you have like a region in the city that has a postal code and suddenly lots of

letters are being sent there and they are like oh now actually we need two post offices then maybe they will divide the region in the city into into two different regions with two different post post office with two different numbers and in practice that does happen right like postal codes change over over long Horizons and in near similarly like okay the whole blockchain is broken down into like these shards and the definition of A Shard can also change in order to kind of Route Around uh the capacity demand in in some way if you think of near to day you can think of it as there

is a uh you know we all observe what happens in the city and how much mail goes to every post office right and at some point we realize that for a particular post office there's a lot of mail coming in uh and it's you know it's getting harder for the employees there to handle it right and so there could be a proposal that says hey guys let's build another post office half a mile away uh and split it like this right and then there there's a separate entity which is validators of the network which which uh choose to you know to go ahead with this change or not and if sufficient percentage of them which I think is 80 uh you know wants to go with this change uh then it's then it happens right and uh you know if you think of

near of the future when the name your sharding is there it's slightly different it's you know as male starts coming into the building another building just spontaneously Pops in without without any human being involved right it's en more like that building splits into too and moves away yeah exactly exactly a new wall appears yes to building separate like on a set of rails and that happens without any involvement from any uh natural intelligence right uh it just it just occurs and then at some point you know two post offices become you know it's getting chill there so they kind of you know come together and the disappears yes yes exactly but but I think the important

parts here right as you said ZIP Coast changes which means like when you're sending your mail you need to like now whoever is in the new Zone need to update everyone they're like in a new zip code you know everybody needs on near you never see the ZIP code on near you say I want a mail to be sent to Ilia right and near figures out the ZIP code itself you don't need to know it that's the beauty yeah and to compare with this with other approaches like subnet rollups Etc I mean in a way they're trying to emulate the same he right it's like oh you know this this rollup is too busy instead of launching there you know you can spin up a new rollup and uh now everybody can go there and you have more

capacity but it's a very you know not just manual and extensive process right I mean like each roll up you know cost is at least a million dollars a year just to run between sequencer explorers rpcs Etc all the infrastructure but it's also now every user every developer every smart contract off like actually trying to use it now at least to figure out how to go there how to bridge there how what gas tokens is used there Etc right so it's it's a huge load on the whole kind of understanding of the network process which we are actually addressing with ch kind of chain abstructure and chain signatures as well because we do believe this is kind of a unit like what we Tred to do with near is a universal problem right is like the

capabilities of the network should be able to change dynamically and every should be able to Route things uh without thinking about the underlying infrastructure but on near we kind of solved it in a very like direct way by having this kind of name space that is common for everyone and using that to run out transaction for messages mail uh between between different participants yeah that is so cool I actually I actually own me.n and and yeah I've never needed to think about what Shard it is on right so to me I I only need meah near

and its Journey Through Time maybe like it was processed in Shard number one then Shard number three and then it's changing and I never need to know about it right like that's that's that's like really yeah I think it it wasn't Shard yeah Shard zero then shard two and then Shar three now it's on probably Shar three right now but yeah you don't you definitely don't need to know about that and like even I am like you know there is a way to look it up but we actually don't show it in Explorer usually because I mean some new explorers show sometimes but because we don't actually want people to know because it's it's

irrelevant information it's like knowing which which exact computer in on which Rack in AWS is you know providing us with this interface we looking at right now yeah yeah so maybe like an interesting imagination for NE is because like ultimately it's like this human memorable names at the bottom and maybe you know like each human memorable name actually corresponds to a person or a company because that's how the world is partitioned and because like the near system breaks into shards you can almost imagine like a virtual collection of people that are transacting their business on a Shard um at a certain point and these

are the users of of The Shard itself and of course like as The Shard boundaries change the collection of people transacting on a Shard is also changing but maybe if we imagine it as like being you know stationary or constant for a certain while which is true for near we can we can think of this like virtual collection of people in the in the near suburb on a sense that transacting On A Shard and then the on the other side corresponding to A Shard you need kind of servers or validators or postmen in our earlier analogy that are kind of processing the mail that's coming to that particular area or Shard I guess like one of the question starts to

become so in any blockchain network you have these validator machines or minor machines that are ultimately like kind of like this postmen or accountants of the system that are doing the processing and even in near I would imagine okay there's a set of validators that are that are found through proof of stake now these validators sort of like need to be assigned to shards that hey you go and process the transactions in in this in this Shard you other one you do it there and how does that process work so I I first want to to correct slightly the first one I think the second analogy was good but the first

analogy was not quite correct because uh even though uh Shard boundaries do not change as frequently today as they will be at some point uh it it it has been the case from day one on near that two accounts residing on the same Shard has absolutely no significance for those two accounts so I think a better analogy for that would be not a post office but like a cell towers right we could be next to each other and I can call you uh and we will be you know uh served by the same cell tower or we could be in different parts of the world and I can call you and know are different cell towers but we have no benefit you know like I I will never know that you're on the same cell tower and I will never care and it is the case on near that if you and I

own the same shart and I send you money or I you know like we transact on some smart contract or we if we are in different shards from the user's perspective there's no difference in uh in experience right so you know fees are not affected the performance is not affected uh you know like you know sharding is completely abstracted to it right and so there's no incentive for example to try to be on the same Shard there's no incentive to grind for example count or like intentionally set the same you know account in the same Shard uh when it comes to the second analogy uh you can think about this way you can think uh I I like I like going in multiple steps and effectively saying you know let's say we're designing a new blockchain uh and we want it to be shed

right and how do we ensure security when not everybody processes every transaction and the first idea would be let's say we have a massive set of validators right so we set the valid you know the minimum fee to be relatively low and we say we have hundreds of thousands of validators or even Millions I don't know uh and then every Shard even though it has only a subset of validators it still has a massive set of validators right so it's a so we have a million total 100 charts and every sh has 10,000 validators then you can say well if if we sample them randomly uh and we relatively certain that the total set of

validators has up to a certain percentage of bed gains right we say we believe that the total set has up to 25% bad guys and not more right then you can do the math and you say well if I sample 10,000 of them uh then the percentage of the bad guys exceeding 33% is so unlikely uh that we can consider to be impossible right to be like you know one over 10 to the power of some large number and then you say well and because there's no more than 33% of bad guys in The Shard we can just assume that they adhere to the protocol that any state transition they approve of like if it has you know as a particular percentage of signatures of those people who are validating The Shard then we believe

that that the transition was valid because the number of bad guys is limited and the good guys sign off so it's a you know good to go it has practical issues we don't actually have a million of validators uh and we do want to have more than 100 charts in uh in the limit but it has a bigger problem which is the contract of a big guy or a good guy uh is very abstract uh at the end of the day everybody who's on the blockchain you know they want to make money uh that's the that's the ultimate goal you know majority of validators I'm sure there are some validators who are there to build the you know the decentralized world of the future where everybody's a happy Corgi owning their data uh but in reality majority of them are there because you stake money so or

rather like you have people delegate to you you keep the percentage you make money right and so correspondingly uh we should think about the security in the presence of the bad guys who will try to corrupt other participants right uh and people talk there are ways for people to talk you know a lot of validators are just sitting on Telegram right and it makes sense for them to be in the same telegram groups because they they run into issues you know like the network is too slow the validator out they need to know that they need to upgrade the validator so they all in the same telegram channels they all easily reachable if the bad guy wants to come and say hey guys I want to do this act of uh being a bad guy uh and I need that percentage of validators to cooperate

right I can DM each of you and say this is how much money I'm willing to pay you because there's something for me to gain from the blockchain gain going down it could be some minor extractable value it could be that I'm you know I'm the uh salana investor and I just want near to go down and salana to go up uh Etc right and so the system need to be designed in such a way that a very large percentage of the validators could be corrupted uh and you know incentivized to do something bad uh and we need to correspondingly let's say we have 100 or a thousand of validators uh and we have on the small small set subset of them in every Shard we should expect that almost all of them will get corrupted right or

even all of them uh and so the system needs to be designed in a way that yeah there are those you know postmen in the post offices uh but it could be that a bad guy enters the post post office gives all of them you know, and ask them to do something bad you know like a route and a mail which was what actually sent by the originator or something like this and so we designed systems in a way that that is prevented or made very very difficult to execute if that makes sense so now I don't know the mathematics of it but if you if you take that that scenario that you sketched out first there's a million validators and then I'm sampling what I mean my sampling is you know it's a huge set and

I'm taking 10,000 and I'm choosing randomly because the blockchain ultimately needs to choose randomly and then if I get the set of 10,000 my my mathematical intution says that if like 25% of that set of million is 250,000 they are malicious in some way and 750,000 are kind of honest if I'm choosing 10,000 randomly my odds of kind of choosing um a set where the majority maybe 66% is malicious means 6,666 are kind of malicious and the rest

a good guys I would think like that would happen pretty frequently right like if I'm if I'm if I'm creating these sets like once every second I would think every six months or every year I'm going to get a set which is which is full of full of malicious no so if you have if you have 250 out of a million then even sampling oneir uh will never happen if you sample 10,000 out of a million then even sampling one third would not happen it's extremely unlikely oh it's extremely unlikely so the so the my my my physical intuition is wrong and if I'm sampling 10,000 out of a set of million and I keep doing

that keep creating new samples once you can do it like billions of billions of billions of times like you can do it every nanc for a billion years uh and it will not happen we actually had the math in one of our original papers right and it was I I don't remember exact exact constants but yeah it was like 10 to minus 20us 30 something like that uh probability so like longer than Universe exists kind of thing but to an extent it doesn't matter because we still uh like if your intuition was correct or if my math was wrong which is possible it would only make the situation worse

right but we're saying we we're saying hey let's say the math is such that and I think it is right let's say the moth is such that it's very unlikely to sample a large percentage of bad guys it still doesn't matter because good guys can become bad guys when they sufficiently incentivized to be bad guys uh and uh and then in the world of blockchains incentives are often such that uh you can you can benefit a lot uh by uh by corrupting you know a set of validators and so you you would every now and then the validators will choose to do will choose to do so right and so correspondingly you want to design a system where even if the set of validators in the chart is corrupted right either either due to sampling uh or because they were corrupted what we

call adaptively right so after the fact uh then the system still operates reliably and and so there are multiple you know there are many ways of ensuring that uh and it's it's the area of research that has been developing and this is where in sharding we we use you know this is the biggest user of zi proofs in sharding because you know if I can just prove that my transition is correct then the problem is solved right instead of sending a bunch of validators if you can just have a prove that says everything is correct then it's uh and the problem is solved but maybe building building this up from the from bottom up right uh I mean it it starts with so kind of we discussed right there's this

account space right you know think of it as a city with people and you know we have the cell tower so when people you know call each other like for for for use of the uh of the analogy when people call each other in the city right uh kind of it bounces to the cell tower and then goes to another cell tower to uh kind of connect them and so so the first thing that we need to do right is to ensure that every second every transaction is recorded right it ordered and kind of a cross all Shard and that no there's no way even for if everybody is corrupted in that chard to be able to

kind of change that order and and for other use cases also you know potentially introduce something that's not valid right and so that's called Data availability problem the near had kind of data availability from uh from 2019 then they desde man shade uh and then you know as other approaches like rollups Etc started becoming more popular they also needed data availability and that's kind of where a lot of the current data Mobility offerings are uh come into market like a a very short primer would be you know like let's use rollups as an

example right so there's a let's say there's hypothetical optimism uh and they they do transactions but they do not use zero knowledge proofs right actually don't know if optimism is planning to use Z knowledge proofs right and so corresponding using zero knowledge proes for fraud proofs I see interesting interesting and so they check in the you know the the the state rout to ethereum every now and then um uh and they say you know I applied all the transactions correctly and if you think I did not I posted just enough of cryptographic uh information of like you know attestation that uh that if I did something wrong you would be able to come and prove that it was wrong right and if you do that I will I will lose a

lot of money and so I have a strong incentive not to do so right and so and so then you observe the uh the rollup and you see that some transaction was applied incorrectly you can go to ethereum and say this is the transaction it was applied wrong and this is the proof right but you can only do that if you actually see the transactions right so if the if the if the rollup was able to operate in such a way that the validators cannot see the transactions then the rollup would be snapshotting something but nobody can prove anything wrong nobody sees what's happening it's like you know it's a fied room so data availability is effectively this concept of ensuring and proving that the transactions that you claim to apply and the state on on top of which you claim

to apply those transactions are all visible to everybody right and and that's something that near had from we were either first or second to have it life I I don't know when I don't remember when PKA do launched and whether they had the data availability from day one okay so you have like a shard and then there's a there's a set of validators that are processing transactions in The Shard let's pause get an intuition of like this set of like let's imagine like Shard two or whatever um the set of validators that are processing Shard two is it like a static set or does it keep changing uh changing with time dynamically it's changing with time so

the idea is and and so there's like what's right now lives and what's been Liv for uh for a few years and also what's we launching with status validation uh I'll probably just start with st about state of validation just for for EAS of explanation so uh with stateus validation there is a two set kind of two roles that the Val that can play and and all of this is uh ratable one one role is so-called chunk producer which is somewhat IL to what you in rollup schol sequencers right so this is the node that receives the transaction uh orders it in the Block in

a chunk in our case uh for responsible for their Shard uh it send out this chunk to others as well and then executes the transactions and receives the kind of the result uh and so importantly kind of where comes in the data availability when when the chunk producer sends out the the chunk information they include they kind of So-Cal the raas code it uh which means they replicate this information in such a way that you know they send it to other cell towers uh such that even if everybody who's in you know servicing this cell tower is

offline ghost malicious Etc other cell towers can completely replicate everything that happened in the cell tower so that's kind of what data avilability eral coding is so there's chunk producer now there's a small set of chunk chunk producers kind of similar how you know there's single sequenc usually on robs but uh you don't want that because of censorship and reliability now for validators actually have a different story right so what Alex mentioned you have this adaptive corruption problem right so if you have validators which are sitting there for a long term for a long time it's it's possible that you go and say like hey if you're in this Shard and I

see you in this Shard for example uh I can bribe you to you know do something for the sh and so and then you need fraud proofs and fraud proofs are complicated and uh kind of require additional timelines and so with with status solidation is say actually the chart producer not just produces kind of the transactions but also includes all of the state required to execute these transactions and so that's so-called State witness and so then any other node in this in a network right can receive this block and execute it without knowing anything else about the network except uh kind of like client of the of the

network so you receive everything kind of you need to validate that if you apply this transaction and you has a state and the state was included in the previous block then the result is this and you can confirm that and kind of send the confirmation and so that's kind of I mean in a way it's uh you know kind of state validation is ability for any node to come in and say like hey I'm ready to validate I don't need to syn synchronize a network State I don't need to maintain the state in on my desk right uh which again reduces the needs for the validator uh node requirements uh they can just like make sure that everything's okay and so now you can rotate this validators

validate what chart all the time every block they can be actually you know you can randomly select set of aors uh they can be overlapping you can select you know out of the million you can select you know 100,000 per Shard and they've Val validate 10 shards for example each if if there's enough capacity uh or or any kind of parameters um and they are able to validate this because they don't need to sink into The Shard they don't need to process every single transaction that ever hit that Shard they only need to look at this specific block so that kind of opens up a lot you know both both kind of you know you can imagine I'm I'm actually really excited that potentially you know not probably this

year but soon enough somebody can open up a new tab you know type in a URL which has kind of a validator node in uh in the browser and actually start receiving blocks in validating right because again like you don't actually need anything else and and browsers haveb assembly to execute the the transactions um eded so that's kind of the like very very light lightweight validation that you can rotate all the time so like I'm imagining this state witness is as more like so let's say like there was Block in and then the transaction came in and the chum producer made n+ one

but is it the case that for every transaction almost like the for every transaction that they processed in in that block they are creating individual Witnesses for each transaction so if you take transaction one they'll say Okay transaction one modifies these two accounts these two accounts had this balance previously after the modification the result is these two accounts have disbalance balance now so our our state witness is this was what was before this transaction was processed this was after the transaction process but the data that's being supplied is only those two accounts that are hit by the transaction

and so for every transaction you are just creating like these um like these bread crumbs these bare minimum amount of info that is needed to kind of validate that transaction and that's the state Witness for that transaction so every transaction that comes in the block of The Shard by this this chunk producer entity is kind of broken down into these individually verifiable pieces and then those individually verifiable pieces are scattered across all of the validators of The Shard and they can kind of do the job of verifying these individual pieces one by one and because each of these piece can

be uh verified individually you that's why you are able to run the validator in your browser because while your browser may not be a powerful machine it can still validate a few of them yeah yeah that's that's right so is it the case that this chunk producer that's kind of like more a more long lived role and then the validators of A Shard is like a more shortlived role meaning as a validator I'm doing some verification in this shart then few seconds later in a different shart then a third few seconds later in a different shart like that I'm constantly switching as a validator but

as a child producer and you say lator is a little bit is a little bit insulting we produce a block every second but yeah it's like every second every block you can be valid a different Shard because there's actually no difference like you you don't actually care which sh it's on because for you it is just a block result like he set a transaction was all the information you need and so that's why you can rotate validators now every second uh I mean there's like networking requirements and some you know data information propagation but in principle yes it can it can rotate every second and then for chunk producers they rotate every eok right now which is 12 hours or 12 to 14 hours uh and there where because you need to actually sync

the state like if you're moving to an next shard for the chunk producer you actually need to know everybody's balances right and so you need to actually download all that make sure it's correct consistent and then kind of while you're downloading you actually need to now receive new blocks and apply them as well and so you kind of doing two jobs in parallel you validated you kind of Chunk producing The Shard you're in now and you're getting ready to produce Shard uh that you are be next time and so that requires kind of more sophistic set up right so okay so then you have these validators that are constantly jumping from shart to sh validating some small

pieces and when is it the case that like when I'm validating a certain piece I'm also adding my signature saying yes I checked it and it's correct and every transaction and its its witness is kind of getting more and more signatures or attestations from the validators and that's how you're building up Trust but it's not accumulating over time all the signatures happen on the same block right so so every block you need to do a sign off uh and it's not it's not the case that everybody signs off every block right uh we we just need a certain majority uh and you know well because blocks are created every every second and the validators are running on a relatively commodity Hardware sometimes

you will miss a signature and like if you go to an Explorer you will see that like like nobody has 100% up time people have 99% uh right but yeah but the idea is that yeah that there's a set of validators they validating a particular block we know whom we expect to sign off on the Block and then you know a majority a percentage of them signs off and the block is created you can look at it and say well that many validators sign off at this point I know which shs they were supposed to validate I know that unless someone corrupted them in .1 milliseconds you know it's a there's a relatively High certainty that the state transition is valid right because on the other side

the mathematics says that when I'm sampling these validators randomly my odds of getting a completely bad set of validators if 25% of the set in the bigger network is corrupt is kind of low so because of of that sampling you're able to kind of trust that okay while these validators are consider or you are sampling them constantly every second you are changing the samples but because the pro you calculate the probability of one sample being being malicious like so 66% extent or something as being very low you are able to trust kind of like the

signatures on the witnesses of your transactions and be be sure of the state of of a particular Shard and meanwhile like these chunk producers when they produc using these blocks they are also forwarding the data corresponding to these blocks to a set of validators now these this this other set of validators may be different from the validators that are uh checking the witness the witnesses of the transactions right they don't need to be the same set and so yeah so so there's two things that happen one is you I mean the the validators that are receiving to check the witnesses they actually received the dat as well

right because they actually can validate the transaction but also you want to send to other shards as well in case you know this whole sharded fail but also like just uh you want to Route the outgoing messages and so we actually combine the message ring data availability and consensus into kind of one process where let's say you have you know let's say like you withdrew money from my account on shard one and then you're sending money to Alex who in shark two so now there's a message going to Shar two saying you know you should debit Alex uh you know 10 near and so now that message is not

just a message it also includes a so-called uh AIG coded part of the transaction data that this Shard uh was producing and so kind of this process ensures few things one is everybody re you know then goes and when they actually signing and confirming their own information and sending the their approval they also confirm they received the needed chunks the needed kind of parts from other shards and so that's also provides as guarantees to this this message delivery from from Shard to Shard it provides the data ability guarantees and it's all kind of you know

integrated into the consensus messages that are being sent by validators uh to each other to actually accumulate their BFC consensus so so it's worth mentioning that there's an next role called block producers right so there's an actual is the blockchain is the near blockchain so it's not like because often when people think of sharding and you know many sharding blockchains do work this way people think of multiple chains right so every Shard is a chain it is not the case on near on near there's only one blockchain uh and and there are block producers creating blocks on the Chain uh and uh when but but those blocks do not contain dexual transactions right or or like

logically they do right like you can think of a block uh exactly the same way uh as you would think of a block on ethereum where it has like a header consensus information uh and a bunch of transactions with the difference that while logically transactions are there physically it only contains information about what we call chunks so one chunk per Shard or rather up to one chunk for shard and physically the block doesn't contain those transactions it just contains the information about the chunks that were produced right and at every particular block some shards might might miss a chunk right because there's a particular chunk producer responsible at every particular moment it could be offline it could be you know uh busy Etc and so the a chunk could be could could

be missed right but if the chunk is produced what happens is that the chunk producer when they produce a chunk as Elia mentioned they rasor coded and they send a part of it to every block producer right uh and the block producer would only sign off on the Block if they have a part of the chunk that that uh that is intended for them and so this is where consensus and data availability are sort of uh manded together is that you know in order to reach against from the block 2third of all the block producers waited by stake have to sign off on it right it's a bft consensus so if there's no two3 of signatures the block you know we wait until we have right we we favor safety over liveness

uh and correspondingly if we cannot get two3 of signatures we will stall right but we uh uh and if you have two-thirds of signatures because the block producer would only sign off on the Block if they have their part of every Chunk in that block right then you know that two3 of the block producers have for every chunk included have their little part and the Eraser code is such that you need one3 uh to reconstruct any chunk right so as long as you believe that no more than one3 of the block producers is malicious and if you have 2/3 of signatures in the worst case all the malicious actors are included in those two3 so you have one3 malicious but you still have one third hus and so you can you can reconstruct any chunk right so every chunk is

available to everybody guaranteed if you have consensus reached on the Block so data availability and consensus are merged together into a single SLE single mechanic right so like this is wicked cool and it's also like hard to understand cuz this is actually unlike any other system where data availability and consensus are usually like like two very separate processes like whether you go from like ethereum to Celestia to all of the but in near it's almost yeah in essence right you you want to it's sort of uh uh begins with with the uh your goals right and then and then we were

going back right so the goal was uh to have cross short transactions and generally uh communication to be with a delay of one block right so effectively if in a particular block a transaction initiated and it wants to do something in another Shard we want that to happen with a very high probability in exactly the next block right uh and so if the data availability was separate from consensus that would be extremely hard to ensure right because we need to be certain that data is available as of the moment when the block is produced as opposed to that being a separate process right and similarly IIA mentioned there are three things which are merge together right data availability consensus uh and the message passing

right so together uh the Chun that is now certainly available at the moment of consensus being reached also contains the messages that needs to be routed to to another shart uh and it is designed in such way that it is ensured uh that by the time the chunk producer of that other Shard is producing the chunk they not only know that the messages exist but they also have them right and so they can immediately act upon it and moreover they have to act upon them right so a Chun producer the Chun would not be valid uh if the CH producer did not act on the messages that were sent from another shark it could be that they don't act upon them immediately because they because of congestion like imagine everybody sending receipts to the same sh right so that is automatically handled right it could be the the

receipt is not processed immediately but the receipt is acknowledged immediately and it's put on the Queue immediately and most of the time it is also acted upon immediately because congestion is a you know most of the time there's not congestion right but it is all part of the same process where we ensure that if something happened in Block n uh and that something wants something else to happen in another sh that something else will very likely happen in N plus one and maybe to to use like etherium and rollups analogy here that you know you have optimism producing a block right that block immediately sends to ethereum validators who

included and uh as well as they include every other role let's say we have our optimism arbit trying to to send money directly between each other right so like both of the their blocks need to be included at the same time immediately in the same ethereum block right to guarantee data Val ability because now uh the kind of our like let's say optimism sense something to arbitrum connecting it but if it's just that right then arbitrum needs to read out the state from from the eum right which adds extra latency and so what

happens here is you assume that Optimus and arbitrum are that those sequences are also validators in the network and so you send them you kind of optim this Mar room send their blocks to each other right they confirm it and kind of those stations are now allowed to progress forward the blockchain and so like as if all the rollups themselves form D of consensus right and we're sending information to each other directly and in turn that allows to optimize for latency and for kind of this cross shot communication uh because everybody talking to each other directly and but rely but then you know sending confirmation to to the whole United

system so I'll I'll try to State this in kind of my own understanding of how it works so so it's like we need two different views one is kind of like The Shard View and then one we need like like the global view because there's a global block so you know we we went to The Shard view pretty well it's like there's a huge set of validators some of them are assigned to The Shard for for a block and then then we reassign to other shards and they are kind of like validating the parts of the transactions in the chart now these validators some sometimes they also get the responsibility of being these chunk producers which are more like long lasting entities for 12 hours for

particular Shard they are they are producing like okay for this block this is the uh set of transactions and here are all of the witnesses witnesses point so these are like longlasting entities so you could have like a shortl lasting role and then you could also get a longlasting role as a as as a validator right so that's kind of like the local view of the shart now in the network there's like a global view where there's actually a single blockchain with like block after block after block like like Bitcoin or ethereum but what that block contains is not the transactions themselves but the chunks from the

different shards that are sort of accepted post consensus as being correct like okay so here's block n it contains chunk chunk one from this chart like chunk X from this Shard chunk y from that chard chunk Zed from that Shard and so on and it just contains what chunks the networks is is considering like finalized and so all of the validators are trying to build this blockchain that just contains contains chunks and the validators are kind of like signing off on that on that block containing chunks they're trying to add their signatures to it and their logic

is something like what they are checking is corresonding to each of the chunks that are part of the block did I get the the slice of data uh in order to ensure data availability for the whole network so as a validator when I'm signing off on a block what I'm checking is okay this block contains these chunks if it contains these chunks I should have received the data corresponding to these chunks do I have it in my hard drive yes okay so that's one sort of validation passed and then I sign and like all of the other validators are signing that so fundamentally the network is coming to

agreement that we have the data that we are going to need to reconstruct any part of the actual transactions in the Block in the future that's the thing people are coming to consensus so in a in a normal like Bitcoin like that the when a block gets finalized the network comes to consensus about the transactions that are processed in near when a block gets finalized network is coming to consensus about the validators collectively agreeing that they all have the data needed to reconstruct every part of the block should the need ever arise is that right

is that intuition right and they also like in the world of stateless validation they also have the information that each of those parts were reconstructed and validated by by a set of validators right right not only that the uh the that the data is there that can reconstruct everything in every transaction in the chunks but also the data corresponding to that every transaction with its state Witnesses was validated by a certain number of validators of the shards where the State Witnesses originated so it's coming to consensus Road data AV that's yeah that's really interesting um really

cool yeah I mean the big the big benefit was and I mean I we do get this question which is like you know why theum uh kind of shifted from sharding to roll up architecture and I mean first of all obviously not talking for anybody individually edium or eum is not an agent themselves either um but practically speaking to design something like this right you need to build everything from ground up you see how consensus data availability message passing and uh kind of validation correctness all layered in into one system and that requires kind of this like approach and as well as the VM

itself right because VM now needs to be aware that compared to eum EDM where every is kind of available right you can always say like hey give me a state of that other account like tell me how much tokens does you know Alex have in in the case of the sharted system like that requires a cross sh message right that requires say like well go find where Alex lives right and you know and ask him how much tokens he has and so you need to design kind of everything from scratch from bottom up with this understanding given the goal we had which is like you know ultimate horizontal scaling that is hidden from the users and developers in many cases

and so for ethereum like they they can do do not have such a luxury right they have a working system extremely valuable extremely uh kind of integrated everywhere and so they needed something that is kind of an evolution of their existing uh system versus a complete rebuild right from scratch and I mean rup architecture you know we use it this analogy because a lot of it is similar right eum provides dat availability and consensus rollups are able to communicate with each other but they need to go through ethereum pretty much to kind of validate and and settle before a message can be passed and so a lot of it is kind of spiritually the same it's just because of the kind of legacy of like well now hm is a whole

separate Universe of accounts right and so now you I as a user have account in every chain right I don't have a singular balance now that they can use everywhere like all of that is kind of a legacy uh you know how do we kind of upgrade the existing system into more scalable platform so that's kind of really the the biggest obviously uh you know inov benefit we had which was like starting from scratch and kind of Designing it uh was this in the luxury of the Clean Slate yeah luxury of the Clean Slate is uh is what you had right right so as a validator sometimes I am taking on this role of being a Long Live chunk producer

for 12 hours in a particular Shard I am constantly taking the role of validating the state Witnesses of A Shard and I'm being assigned from Shard to Shard to Shard and then every time I kind of like a block is produced in that Global main Network what I'm signing off on is is corresponding to that block I should have received some data to back up the data of the state of the entire network the data of the entire network I should have received some small chunk of it I can identify it have I received it and then I not only should I have received data but I should have also validated

some of the Witnesses in the global Network or in The Shard have I done that and if I've done that I sign off and if a majority signs of that is when near says okay we have consensus overall that the data is backed up properly and all parts of the update had Witnesses and they were validated by enough enough um enough validators and therefore like this block is correct and like then that's done and then the chain moves all yes yeah that's right and at the end of the day all of that complexity boils down to you know like there there are checks and balances that people do but

at the end of the day all you care about as the Observer is do I have signatures from the sufficient set of validator and if I do I know that uh they have done all the work or at least they claim to to have done so right but that's but that's a whole set of validators right so for them to be corrupted you need you know you need to corrupt a uh like a whole big proof of stake system which uh you know yeah it has C its own you know design there are certain design principles that don't allow it to be corrupted without a massive amount of money being lost uh and yeah and so that allows also you know other chains to easily create light clients right so so near itself

encompasses a lot of processing internally it could also be a part of a bigger ecosystem because building a light L for near is a relatively straightforward process and near having you know general purpose wasm machine can can run any light client for any blockchain that allows a light client and that allows you to have uh two directional bridges that effectively say as long as the light client of both chains is not compromised and light clients are very hard to compromise despite the term light client like you know compromising an ethereum light client or near light client is extremely hard right and so then you say well for as long as those are not corrupted near is also part of the bigger ecosystem of of more chains so I guess like every creator

with when they make something they end up having like there usually certain element of the system that they wished was better and then do you have those like what what for you individually what parts of near are you kind of unsatisfied about in the design well I mean there's few things that we're still finishing up or like and and still on the road right I think so we mentioned Dynamic three sharting right so right now as Alex mentioned right this is more of a kind of a governance technical governance process but the benefit with the stateless validation approach is that because we rotate validators you know every second now you can actually change the number

of shards for validators very quickly right because again they don't really care how many shards in the network they don't care which sh Val they just receive the B and so the benefit here if we have sufficient kind of redundancy on chunk producers we can split the chunk producer group and say like you know as of next block you don't care about half of The Shard that you were in right so comparative you know to to where we are now where we need to like everybody agrees you know new client get been run validator spit it up you know re shotly happens this can happen literally instantly where everything like okay now we split into two shards you know now

I'm ignoring just H and transactions that that may still receive because people to routing to me and I'm just processing this half and then on the junk prod production side and then on the V side you know now you just assigning yeah sampling just from a different for for The Listener this is the cell tower splitting into two dynamically or the post office splitting into D exactly yeah and this it can happen like immediately right uh which is you know hu huge uh benefit for this spike in usage where you know some application just went you know viral n you know to claim or whatever and you can like hey let's just pull it out into a separate Shar let it you know go nuts and then maybe merge it back uh in like a day or so so that that's that's you

know a definitely huge kind of benefit thatb and and for context I mean status validation is something we published um in February I mean been building last year and uh should be launching within the next few months and then we're going to start working on dynamic now from my perspective there is uh few few things that uh beyond that uh that would be that's we should be working on one is and and Alex actually worked on some the volumes of this ear is uh leaderless uh chunk production so right now there's one Chun producer at a time that is responsible for producing chunk

right and I mean they kind of wrote dat and they randomly ass sign but still uh you know if that chunk producer is offline now we have a gap on the 10 slot and you know you drop in GPS you have em know High latency with the users and so the idea definitely is they can also be they can also be deduced which is uh yeah something that happens other network like out no not not because we somehow resilient to that it's just bad guys chose another Network to to get do not ours but it can happen eventually right so so liter CH production and consensus is something we need to implement and and we have a pretty clear way of achieving

it so Leader by leaderless chunk production So when you say that M kind of like reminded of algorand where where the the essential idea is like when you think of a network like like you you think of like a cosmos Network the blockchain is rolling along end blocks have been produced everybody knows who should be producing block n+1 right it's publicly known and if they fail to produce the network Waits and then realizes fails to produce and then the network also knows if that guy fails to produce the block then this other guy should produce the block and

there's kind of like a almost a line a cue made of like who has rights to produce and so that's prob what you mean when you were saying leader so if you have a Shard you have multiple validate like uh multiple chunk producers but which one exactly produces the chunk for this block it's currently known in near just like Cosmos but what you would like is a system where there are n junk producers and then when a certain block rolls along one of the chunk producer realizes they have some kind of winning lottery ticket and they can produce the chunks so this is no this is not so

you're explaining algorand and it is not leing I'm explaining algor so is is is that division for lead no there's still a leader in what you explain so the only difference is that the leader is not known in advance right it partially solves the problem it's much harder to gidos for example you cannot gidos someone you don't know right and so by the time you know that they want the lottery ticket uh and we actually do uh the way we think of it uh is similar so an algorith you actually don't know in advance that you want a lottery ticket you you know like like effectively what happens is that everybody looks at their lottery ticket and sees the number and the highest number wins but you don't know if you have the highest number because you don't know the numbers of others if you did and others did then it

would be no different from Cosmos then everybody would know right so instead you say well I have a sufficiently high number for me to believe that I might be the winner so I will produce the block I will publish it maybe someone else will publish and you know given my number was 97 out of 100 uh there's a good chance that I was the highest right but maybe there was 98 the this approach has has a minor problem that still multiple blocks will have to be broadcast right because well like effectively either the threshold is too high that every now and then nobody will broadcast a block because nobody want the ticket above a certain threshold or it's like sufficiently low that multiple blocks will will be broadcast uh but in but in our case something interesting to think about is that at the end of the day the

the chunk will be broadcast to everybody like a little part of it right and so everybody will have to receive but within the network within the uh within the set of the chunk producers what they can do they will have to send the Chunk in its entirety for validation or like with with the validators of the of The Shard right so you can think of a system where on the high level what happens is that every single CH producer generates a chunk not just some people who who are Beyond sort on threshold everybody produces a Chun and then everybody stands to every person in The Shard so like to every CH producer every uh you know like uh ER recoded part and so at the end of the day the network overhead is not chunk

size times number of participants it's still proportional to the chunk size right but uh but now every but now you have as many Chun as you have ch producers and then you still do the lottery ticket then you reveal your lottery ticket and if you want your chunk is is accepted right but there's no issue with the uh treating the threshold and maybe spamming the network with multiple chunks that have to be exchanged in its entirety so that's the high level idea right but the high level idea is that now you can literally never a chunk will be skipped uh no matter you know like as long as like there one person you didn't didor the chunk will not be skipped and it's a it's a slightly slight slightly improvement over an algrant idea where you you know have a threshold and uh but still

exchange the whole block yeah this is really this is really fascinating so I think the essence of the design being that in Bitcoin or in ethereum when when I have a block I have to forward that entire block to everybody and there's a certain like diameter of the network so I'm a validator and there's some validator to which I have the worst connection like there's multiple Hops and that entire block must be now sent across the diameter of the other side to that worst other validator but in near almost it's like I have a more

efficient um microphone or megaphone so I produced a block I cut it into lots of pieces and I only have need to send these small pieces to all of the validators so there's some validator to which I have the worse connection but I only have to send a small piece to through that worse connection through that diameter of the network and because of that because because like this this broadcast of is only piece piecewise and this broadcast is kind of like efficient you can afford to have a design where in A Shard all of the chunk producers produce a block they broadcast their pieces and like then

they compare okay which of these pieces has some highest amount of Randomness and that becomes like the canonical chunk for for that block yes it's got be chunk out the block yeah and it's like depending like that person I have the worst connection to depending on why they need my block right if they just a block producer and they just need to to sign off uh it's sufficient for them to have just that little piece they don't need to reconstruct the block or a CH sorry a chunk but if they do need the whole chunk it's still more performant than sending a whole chunk through some right like I send little pieces and then that person will collect little pieces from different entities right so it's still a faster way to propagate

information and and I guess the advantage here for us in building that leaderless consensus sorry leaderless chunk production is that we already have a concept of this eraser code right we already send small pieces we already have the mechanic to gather them right so that's uh it's much less invasive change uh than for a network where today blocks are being sent fully right so for them to implement the same feature would be implemented a lot of uh new mechanics while in our case it's just plugging into existing Machinery I I wanted to touch so you asked a question which I think is interesting and and IIA covered it from a very different perspective uh of uh sort of you know uh uh

a starting that uh design wise Is Not Great uh and we would like it to be different I I think something uh I thought a lot from the day we launch near is that the accounting model for sharding is quite different right like you cannot have uh like fles loans for example easily because uh accounts are on different shards and everything takes a hope right and we've been trying to solve this problem like we were trying to find a way to have Atomic transactions since day one with many different designs uh and it's a drawback of sharding but what's interesting is that I think slowly we're coming to to realization that long-term not having sharding is not an option right so in essence the highest put

blockchains today which are not sharded are getting congested and that it's only that much they can squeeze more right like they they work day and night to remove all the suboptimalities but at best they will squeeze out another 20% right let's say 50% that will get congested again like the adoption of blockchain today is a fraction of what we want it to be to consider the whole ecosystem to be successful right and so sharding will have to happen and when sharding has to happen people will have to deal with this disadvantage AG of of this different account model and so near in this case is positioned extremely well because from day one every application near was built with that account model in mind right so we have tools we have understanding developers

in the ecosystem are used to working in this thata PR while in the rest of the ecosystem people are still operating in this uh Atomic transactions mindset which they will have to abandon eventually because at some point if their application is to scale and the applications they depend upon are to scale they will not be able to maintain this Atomic guarantees uh and scale to the usage right so they will have to abandon this mindset uh and we're positioned uniquely in the sense that everything that is built on near this this you know reach ecosystem of applications is built in a future proof way you know as we create more shards every application near gets to take advantage of that uh while for any

application that is built on uh you know what we call synchronous uh runtime that they they will have to to rewrite their applications I think that actually the really interesting part is because here is kind of a synchronous and in the way every contract every account and every contract is designed to be independent it actually doesn't matter if that account is on near or not and so that's why kind of a lot of the chain kind of abstraction idea has been coming because well it actually doesn't matter for like for rest sign like an aml1 Mir doesn't matter if the token is on near

onium on base on salana like you can actually deposit any token and the rest side ances will able to handle it and so so like generally speaking near is design with every like every contract that we count is kind of handling assets that are living on like in a synchronous way right and obviously it's good to have them on near because you get like one second communication time so like the latency is very low there you know that count spaces is nice but it can be living somewhere else and if like if we have kind of a message passing way of sending this then you know near smart contracts know how to

deal with that and all standards like here C20 analog standard is designed with callbacks and kind of Messing mind and so again like compared to right now inms they trying to figure out how to do cross Layer Two communication the challeng is really lies in like all of the standards like imagine trying to send nc20 from one chain to another like there's no Standard address space or anything that supports that it's also synchronous like expectation is that that transaction will execute same block but actually it needs to you know be scheduled message pass settled you know send somewhere else R valid Etc so so really that's kind of the it it it was a very non-trivial tradeoff right and and

we kind of I would say had had a period of time where we're like did we do a right tradeoff kind of um but but right now yeah we seen like we're literally seeing the validation of our Theses kind of throughout yeah so the point being like something like flash loan where in the near design it's hard it's not that flash yeah it's more like iron and loone yeah H yeah it's not that flash so at some point in 2021 or something like that it must have seemed that oh ethereum has flash loads but near Wood and that's a problem or is a perceived problem but when you look at like ethereum in 2028 which is ethereum plus

all of these l2s at l3s you can't have flash loads across that entire ecosystem and you can't have flash loads across near so it's fine right like it's a it's a downside but not really exactly like the modern D5 of 2028 will be what what people have been building D5 near for the past uh two three years and Asia mentioned this entire ecosystem of l2s and l3s will also be part of near ecosystem because because of chain obstruction right any account on any chain can be you know with a very thin abstraction layer perceived as a near account just with a higher latency right

so so like a near account to near account will be one second always but near account to optimism account it's going to be exactly the same protocol on near s side but the latency will be higher because there's a communication between them yeah I'm not going to chain abstraction because I feel like we've already covered so much and I did Cover chain abstraction with in the previous episode with eara so I mean avoiding that cuz things things start to become only to compx curious listener can Google near chain obstruction yeah I just wanted to to to give an understanding why like like chain obstruction didn't come from nowhere chain obstruction was the

mentality we took was near when we designed near it's just like now we expanded that to kind of all chains but it's still kind of you know the thinking with put in into designing near is still there kind of now how do we apply it to the whole web three again assume like you can sync of you optimism just being another Shard of near and this is where actually like you know for example Z proofs it and agir what aligon is working on is all coming together because like if we can unify security if we can kind of provide kind of common security layer then on top of this you know and again we have near da we can actually settle da of other layer twos

well now they're actually not that different from other shards of near I mean there's differences in sequency and production and so like there's things to handle under the hood but again we can kind of extend our layer of obstruction that we provide to users and developers to kind of cover up that and say like actually you know if there's DK proofs and data availability we can actually like say security is the same and now we can message past and we can do kind other species this way and so so that's kind of the idea is like you know how do we how do I apply the same methodology and and kind of user experience developer experience but uh but then expanded back more uh to the rest of up three so like from my perspective you

know when I look at like ethereum ethereum's road map and then like the near and near road map one of the things that stand out to me is in ethereum the road map is based on like scaling via these l2s and l3s but the relationship between ether the asset and the core asset of the L2 can be synergistic in at times but it it can be non- synergistic at other times right like so so it's like the L2 pays the main ethereum chain for certain services and the service intended is usually data

availability but it can be the case that um the L2 generates 100 million in fees but it only pays 500,000 in fees to the main chain it's this this relationship is kind of like great if the L2 is kind of uh building a completely new market that ethereum never had right like imagine I don't know some some I decentralized app comes and the H to capitalize on it built it uh they got 100 million in transaction fees and paid 500,000 to ethereum main chain it's great for the ethereum main chain cuz there's a new Revenue stream coming it 500,000 but it's new the interesting case becomes

when some app that was massive like that's like massively popular on the ethereum main chain generating millions in fees ends up thinking it's better I migr to that L2 and so they might be making like 10 million in fees on the main chain and then they migrate to the L2 and then the fees get cut to a million and then ethereum is only making 100,000 on it so so it was making 10 million in fees and now it's only making 100,000 in fees because the L2 ecosystem exists and there kind of like the relationship is well from The ethereum Ether holders perspective that isn't so ideal right

because you're you're losing a Dap that might have been cultivated by the ethereum network over years and then now it's kind of like migrated away and in practice this has happened with something like dydx but what's really cool about near is like this sort of system doesn't exist like the shards like the relationship between like near and The Shard is kind of the near token kind of like alls the revenues made by all shards and kind of uh there is not in order to scale it doesn't need to have these complex economic games be present um between kind of like a main

chain and an execution layer which is very much there in ethereum and I believe that this will be this will become a relevant feature of ethereum's ecosystem politics in the future and Nia will just not have any of it cool so yeah I guess we can we can keep it at that and it was uh it was great to have both of you uh on the podcast um maybe we should have another one to discuss on how Alex is planning to use recent developments in AI technology uh and what he's building there I was going to say that it's not the coincidence that AI stands for Alex

Amilia yeah I mean thank thanks for having so obviously this is a highly technical topic that uh I think it's it's been really hard to explain in general I mean we've been trying to do this for years now but the kind of the core idea is also like it was really hard to prove it out when you just launched because when you just launch you don't have anything so there's no users so there's no need for sharting and I think like we going to have that problem in web 3 where everybody was claiming scale but but until you actually have like real world you know massive user base to actually transact you know just like a a general Improvement uh was enough and I

think only in last probably like 3 six months we're seen you know a near for example kind of multiple kind of million user application launching right near right now somewhere between 1.5 to 2 million daily active right which is uh more than uh uh any other blockchain right now we have more transactions usually than all their two combined at least on Sunday and so like that that's kind of where this is starting to prove out right um and uh for contact we're still under salana transaction numbers so salana counts salana counts the consensus transactions

so it's a I don't know how we compare to the to the true member yeah but but generally speaking Yeah like we we have more data active users than salana and most of the days Tron because Tron is kind of second the biggest right out but again like this this is the point that you know as we started to see can of's growth like we started to see you know congestions of s the shards and the idea was that you know we can just expand capacity without increasing fees versus every other blockchain including Tronics really good ex example because the transaction fees went from being very cheap to actually now being like you know 30 50 cents uh

for their users even though they are running you know with like aess small subset of validators you know a a kind of modified inm uh chain so I think like that's kind of where we starting to see this things play out and obviously again it took a while right to ecosystem to mature the applications to to you know build and launch as well as for them to gain users but now uh we're starting to see this uh story really play out and you know on thisly uh it's it's exciting and it's also kind of tell Now is really good time to tell the story and kind of explain how it

works cool then I'll catch you again on epicenter IIA and Alex thank you thank you for being thank you thank you