Sui Lutris: Combining Broadcast and Consensus in a Production Blockchain, George Danezis a16z crypto

Transcript

uh welcome everyone to this morning's uh a16z crypto research seminar super happy to welcome George Denise's um he's a really well-known uh computer scientist professor at UCL and also Chief scientist at mistin labs and he'll be giving us an overview of sui George all yours thank you Tim and it's a great pleasure to be here um I'm going to present uh sweet lutras which is the the core distributed systems primitive uh underpinning the sweet blockchain that just launched a bit more than a month ago and this is the culmination of many years of research of people both who are working at mistin Labs as well as uh wider

collaborators that we have worked with before uh I'm indeed George denezes I'm a professor at University College London and chief scientists at mistin labs and to some extent the the underlying design is uh part of my work uh for you know over over five to ten years so let's Jump Right In uh what makes a blockchain is is the first question right to have a blockchain you really need a lot of components uh you first of all need a kind of strategy a protocol for doing distributed replicated transaction processing so so this is actually the kind of boring end of blockchains if you want uh like transaction processing is as old as computer science you know it's

it's the kind of underpins most things that companies and organizations do databases Etc right uh and and they're also kind of underpins to some extent uh blockchains right I mean a blockchain that that has smart contracts to some extent does distribute a transaction processing that's what it does now of course blockchains are much more than this okay so to have a blockchain you also need to have some strategy to make sure that you have civil resistance to create a permissionless system you also need to have tokenomics incentives so that uh you know you don't get flooded you don't have denial of service attacks everybody has incentives to act honestly Etc you also need to do a lot of crypto you need to have high integrity data structures Merkle trees there are

knowledge proofs Etc and if you want to have a blockchain that really can be used for all applications you need to have some strategy about privacy because blockchains by themselves are public ledgers effectively I'm not going to be talking about any of these later Parts I'm just going to focus in this talk about how do we ensure that we can do safely and reliably this distributed replicator transaction processing quite efficiently at high throughputs and at low latencies this is what the the topic of this talk is going to be today sui as a blockchain of course does quite a few of the other things but these we will talk about another time so let's first examine what is the the kind of most reliable and I would say

the dominant Paradigm even in blockchains for doing replicated transaction processing okay and this is called State machine replication okay and I I would really love to talk to the folks in in the uh in the 80s that that really coined this kind of um pattern for doing it because I I suspect that before this pattern was really established uh there was a massive mechanism that led to all sorts of unsafeties and you know lack of liveness so this this work uh on state machine replication from the early 90s late 80s really coined a pattern for how can we do safely uh replicate a transaction processing and largely speaking most

blockchains today kind of follow it which is a testament to to how Timeless this is so so the flow goes a little bit as follows and and is Illustrated in the slide transactions come into the broader distributed system okay and because transactions usually are not sent to All participants of the distributed systems they go into a mechanism that we usually call a mempool in in the domain of blockchains that kind of gossips and Distributors transactions around the miners or the validators of the blockchain okay uh this is usually a peer-to-peer Network and then there is a distributed protocol that broadly speaking sequences those transactions okay so now we go

from this world of a mass of transactions that is just gossiped around to a world where all the participants in the blockchain the miners the validators whatever actually have a common sequence of transactions okay and then once basically they have a common sequence of transactions we go to the next stage which is execution of those transactions so now all the components of the blockchain the miners or the validators deterministically and in the same sequence execute those transactions using usually some kind of isolated execution engine beat the ethereum virtual machine sui uses the move virtual machine but you can use other things like the

Berkeley packet filter or whatnot there are many options there and then once this execution happens and concludes there is a resulting State and many blockchains actually summarize that state and also authenticated cryptographically and basically put both the transactions and maybe some summary of that state into blocks and kind of send the blocks around uh for the purpose of archival and the purpose of dissemination the purpose of basically updating everybody about what happened at the end in in the blockchain now some blockchains do these things in slightly different orders in particular the longest chain uh designs kind of

start with blocks seemingly and then build everything else but broadly speaking you would recognize most of these steps and I myself have designed quite a few blockchains uh that follow this pattern I was involved in the design of for example the Vega protocol it uses tender mid for doing sequencing and broadly speaking follows this pattern I was also involved in the design of DM that basically follows pretty much that recipe and and they work fine there is no no fundamental problem but despite the fact that there is no fundamental problem there are all sorts of problems that still need to be solved right and and and this is really where building distributed systems is truly one of the most difficult tasks that as a computer scientist I have

faced because there are all these details that are quite critical to both correctness and performance that one has to actually take care of so on on the front end of a blockchain in the mempool uh really it's a world of chaos at that point transactions have come through nothing really is authenticated they're running really very solid stories for doing denial of service protection so it's a world of like heuristics uh where there is actually very little research on how to build these things in a way that is correct and scales it's largely uh a set of ad hoc designs and kind of engineering heuristics there and there are fundamental problems because a lot of resources and time is spend in the dissemination of these transactions in

the mempool so if you get that wrong and for example tender mint in one of its variants did get it wrong the whole system just doesn't work right so and and sadly there isn't actually a lot of research on how to do this right then if we move to the next uh the next phase um the the sequencing and is is where basically most of the research has gone into in this famous consensus part of blockchains right in order to make it more performant and and more reliable as well and and still today I cannot say that uh sequencing is totally out of the woods for example in terms of performance it takes many seconds to many minutes for for longest chains um and traditional sequencing mechanism

consensus mechanisms had kind of low throughput uh and are quite delicate and hard to build as well so it's quite a difficult thing to to to build uh and then even though this is difficult to actually build the sequencing is difficult to build and and get uh to be performant traditionally the execution is still a bottleneck even even though sequencing is slow uh so virtual machines that execute uh ethereum for example can do a few hundred transactions per second in a sequential fashion uh the execution is not just slow because of the virtual machine but it is also the interplay of executing over a huge state that you have to keep accessing so there is an interplay between getting the transactions in in a sequence executing

in sequence pretty much on one core and reading and writing from a database in a kind of like sequential way so all of this is very expensive and and not super well optimized and finally many blockchains when they are about to form their blogs need to maintain high integrity data structures such as Merkle trees now a Merkel treat when you implement it in your crypto one-on-one course over like maybe 10 items or 20 items is a trivial data structure it's just a tree with like at every node that you you summarize in a root when you actually have to build it over a database that spans terabytes it is very difficult to build in an efficient way right because you have to keep doing these random reads across your storage and you can do tricks like keeping the

top of it in memory Etc but still it is quite an expensive data structure to maintain so a lot of the um uh uh you lose a lot of performance if you basically don't do that right so as you can see every single step of that process is delicate is critical for security and reliability and potentially is a performance bottleneck and this is the challenge that basically we're facing where we're building a blockchain how do we solve these problems the pattern is well understood but how do we solve these problems and in my own research over the years I have provided answers to some of those problems so we as part of the DM project actually we looked at how do we formalize mempools and how do we actually make sure that

the data dissemination there is both secure and extremely efficient and then we we leverage those kind of designs in in narwhal as we call that system to build consensus on top of it which we call task and and there is an asynchronous variant that is called bull shark so to some extent we have ways of doing high throughput now sequencing and and correct kind of mempool designs and and then we looked at the next bottleneck we looked at execution and we thought well why not use all the cores of a computer in order to do execution and we designed the the block STM algorithm for doing that but there are still some important questions that were remaining and answered that we try to

answer when we designed Suite all right and the most important one is how do we make sure that when the capacity that is needed from a blockchain exceeds if you want a single machine we can still scale up the system all right and how can we actually ensure in all cases that the latency is even lower than the latency we can do for processing transactions if we use consensus as I said despite our great advances and consensus and even in in bull shark or or other consensus mechanisms like hot stuff the latency is a few seconds this is not very satisfying for a lot of applications can we actually lower it even further for

common classes of of transactions at least and this is uh to some extent one of the main motivations of of why we build Suite the way we build it now as I mentioned State machine replication is a very reliable Paradigm for building uh blockchains but there are also other paradigms okay and we're going to lean on some of these paradigms to actually build low latency processing paths in sweep and one of the most interesting uh findings if you want over the last few years is that you actually strictly speaking do not need to have consensus

to build the cryptocurrency okay this was mind-blowing when I actually started thinking about it and I started reading other people like a Rashid guru's work that pioneered this approach okay because if you think about it uh we we usually use consensus in a cryptocurrency in order to avoid double spending right so if someone sends a transaction that spends a coin conceptually the first transaction goes through the second transaction that uses the same coin is void and just gets annulled but in order to actually guarantee this property you don't really need to sequence everything with respect to everything else you only need to sequence transactions touching a

particular coin with respect to each other okay and this is really the the insight so since a coin is owned by a particular party right by particular uh you know owner that that that owns the address of that coin the the inside of that line of work is that the entity that is the only entity that can spend that coin effectively can suggest a sequence of operations and the system just checks that that sequence is correct rather than actually coming up with a sequence by itself and it turns out that in order to do that you you only need a much weaker primitive than consensus because someone already suggests the sequence and the system just checks it rather than the

system having to to decide what the sequence is this primitive is known as consistent or reliable broadcast okay and consistent or reliable broadcast broadly speaking has one initiator of the broadcast sending and otherwise has replicas out of which uh at most a third are are corrupt uh and the initiator sends a message and correct replicas eventually deliver a message that is broadcast and and informally this uh consistent broadcast or reliable broadcast they are slightly different but they offer informally at least two properties the first property relates to safety if any correct replica delivers a message

and another correct replica also delivers a message the message is guaranteed to be the same okay so that solves our double spending problem right if basically on each coin you can basically broadcast who I'm going to spend it with and one replica here is that I'm spending it with Alice the other one will also hear that I spend it with Alice so that's the kind of safety property and the second property relates to liveness namely a correct initiator is always able to actually make the protocol progress from initiating uh the broadcast to all the correct replicas actually delivering this message so that gives us liveness so what is the catch what is the catch right and the catch is

really a subtlety that I mentioned in the second property namely a correct initiator can guarantee to make progress if the initiator is incorrect or Byzantine and the equivocate for example the protocol is not really guaranteed to finish okay so so that is something that we will have if we use this primitive to design around to some extent right to to to somehow see how can we expose this so that this lack of liveness in this Byzantine case does not restrict us too much in in what we can express and in fact uh when when we we started looking at this work back when I was at Novia already

um we designed a cryptocurrency that uses consistent broadcast in order to do payments effectively I mean Novi at the time for those who who don't don't know the history I guess uh was the Facebook division that was building this famous cryptocurrency Libra or DM and the vision was and some of us know it already that one day it's going to be used by two billion people to do payments Etc right so it had to scale right and and this was one of the ways we we thought it had to scale and be low latency so we started thinking about these consensusless agreement protocols in order to to to scale it up and and we designed fast pay uh fast pay

I think was actually published at aft 2020 that Tim you probably uh were cheering at the time I believe that that was one of the code ones I I don't know because I didn't go I sent my students to these things these days so so fast pay was was actually a very uh a great system if I may say so myself right you know we used multiple machines and multiple cores because it turns out that consistent broadcast is highly parallelizable and like consensus um and we we were getting transaction processing for simple payments from you know 40 000 to 160 000 payments per second it had extremely low finality

because these Protocols are literally a kind of one uh sorry two round trip protocols right so uh we were getting 200 to 300 milliseconds finality so it was not bad at all as a protocol it was a very good protocol but um it had some limitations which is that it was just limited to payments right so the only operation you could do is the kind of like simple Bitcoin operation of taking account an account has you know an address and a sequence number and a certain amount of coins in it and the only thing you can basically do is transfer you say okay from that account this address and that uh you know version number take a bunch of coins and

send it to another account that's it right um so the question that we we had when we started working on on sui was can we generalize such a fast path 200 milliseconds such a scalable path right to General smart Contract Processing right not just payments and if we do how do we solve the problem of just having a system like uh fast pay which is extremely asynchronous since we're not using consensus things happen in parallel things happen in parallel on different cores and happen in parallel across different cores how do we actually then make all the other functions of a blockchain work

such as for example the need for people to have an auditable kind of history that they know is complete eventually right or how do we actually allow operations on uh accounts objects something that are owned by multiple people that are not cooperating with each other in order to suggest sequences of actions as in in consistent broadcast and how do we combine maybe fast paths with you know slow paths that are needed in order to do that uh and also how do we do committee reconfiguration the the fast pay mechanism did not really tell you how to change committees ever in fact it was a side chain of of another system that

that was doing that and I mean the question of privacy also came to our mind but we are not going to talk about this today by the way since uh since I'm talking here I would like to point out that one of our co-authors uh Matthew Bode is actually going to Market with variants more powerful actually and more General variants of fast pay uh linear uh and uh he also has some answers to those uh those questions which are very interesting by the way they're slightly different than our answers of course so I guess you you will have a talk from him too uh to find out what he he does uh extending this work so quick question so you mentioned sort of the weaker liveness guarantee of

reliable broadcast so like in the payment system that just results that you might find your coin Frozen basically like unspendable that's right that's right so so in the payment system it's a straightforward world right each account is owned by one address uh one address is one correspond to one signature key and one entity has it so to some extent that entity has to sequence correctly if ever there is an equivocation the coin is locked right forever so this was also a problem that we we wanted to solve this is not very satisfying to say that okay all your funds now are forever inaccessible so that was an additional problem that

we're solving here yeah well good all right so fast pay offers fast paths but is restricted traditional State machine replication offers a generic way of doing any transactions but is slow this is really the equation we have here right and now what we're going to ask ourselves is how do we combine the two okay how do we combine the two to build a generic smart contract platform to make sure that you know we can do something fast something slow everything is safe then we can combine all of this in ways that you know are consistent

with each other right that we can execute in parallel we can get early finality as soon as we can as soon as it makes sense and that we can also have functions in the blockchain that you need in a kind of like production blockchain such as checkpointing and reconfigurations that we can change our committees this is really what we're going to try to do and also down the line how do we build an architecture so that we can scale each validator to be effectively multiple machines so we can support any kind of demand that comes that comes our way and all of this has to have happen while maintaining safety in the system the more we talk about things happening in parallel in terms of execution in terms of distributed systems in terms of many machines the

more there is room for inconsistency right so we have to make sure that all of this is done safely okay and this is effectively the the recipe or maybe the the problem statement I would say that led to to the design of The Suite blockchain so let's look at um how does a validator in in Suite lutaris which is basically the underlying mechanism of sui actually operate and it is quite different I mean quite different it is an extension I would say of the the state machine replication and combination with um the fast path so so what happens is that users submit transactions at the far left of the slide and and the first step is that a user sends the

transaction to all validators and all validators check that transaction and I will I will talk about what check the transaction means and um send the transaction back to the user this error that goes from the transaction to the consensus is a mistake and I will talk about it uh then the user gathers signatures that validators have sent back after they check the transaction and gathers them into a certificate what is a certificate is effectively signatures from two-thirds of the validators by stake in the system okay so now we have a certified uh transaction which we call a certificate and that certificate is again sent to

all the validators in the system okay and now what they do is they check that certificate and we will talk a little bit more about what check that certificate means and then and like the the line that is actually on the slide the certificate when it is checked is always sent to consensus but there is a fast path under some conditions where we can just go and execute straight away and this is really the the key path here that allows us in many cases to have very low latency after the certificate is checked we also acknowledge the fact that the certificate has been received and is valid and will eventually be sequenced

executed Etc to the user and at that point we we get what we call transaction finality and I will talk again about what is that guarantee that we get immediately basically after two round trips here in all cases now if the the all the certificates go through the consensus path uh the the arrow should actually be that way uh validators talk to each other through the consensus algorithm we're using which happens to be a narwhal bull shark and then eventually we assign versions to them we will talk what uh what that means and we execute the ones we couldn't execute already and we store all the data in any case and then when basically certificates are both uh

sequenced and their results are known the results of the executions are known we are ready to uh form checkpoints and send them to all other validators and archival nodes Etc and this offers the kind of like reference sequence of the system however as you can see the checkpointing kind of happens asynchronously already when we execute and restore either because we have followed a fast path or because we're following the consensus path we reply to um to the to the clients that send the transactions with the effects of their transactions that are assigned and there we get settlement finality which is a stronger kind of like guarantee not a

finality but but allows you to do uh more things than just transaction finality so this is this is the the flow of transactions within a validator in uh in in sui lutres and the thing to note oh sorry the thing to note is that this kind of red box that encompasses checking the transaction checking the certificate executing and storing effectively is the fast pay path right it is a consensusless uh path whereas the path where we receive transactions if you want we put them in

a consensus we assign version then we execute and then we checkpoint is effectively the state machine replication path so sui looters is this kind of Frankenstein system effectively that just takes these two paradigms glues them together but of course because some things may be executed through the fast path and then have also other things that are executed through the consensus path we have to make sure that they're consistent with each other now let's not be abstract about that let's actually be quite concrete what if I have you know some piece of state that is referenced both by an operation that goes through the fast path and an operation that goes through the consensus path and it uses let's say the

same coin or the same object or whatever something inconsistent how do we ensure that only one of the two things happen right because we we cannot have both things happen and then have double spending for example just because we use different paths okay so that is the the the the the difficulty that we will have to deal with and similarly as you can see here we have a very fast path right and we also have some checkpointing that really relies on the consensus but how can we ensure that for example when we close our epochs when we do reconfiguration whatever when we get finality through the fast paths we can guarantee that eventually things will happen and will

be represented in checkpoints and will exist across epochs again there is a potential here for inconsistency uh because of this fast path versus the the consensus path that is leveraged in order to build uh all our checkpoints Etc question to what extent is it helpful like should I think of this as almost like a black box composition between like a reliable broadcast protocol and a pvftr protocol or does it does the correctness kind of rely on details of how those components are built so it is absolutely a black box composition so in here lives a consensus protocol just in the consensus and sequencing of certificates box and on purpose

we have chosen to expose nothing about this protocol to anyone but validators and and this is very important to us because ultimately we expect to make fundamental advances in in that space so and like most blockchains that really try to in effect structure everything they do around their consensus we're actually trying to hide as much as possible about our consensus so that literally we can evolve the consensus from one day to the other without anyone in inconse in the ecosystem being bothered by this except the validators whose job it is to be bothered by this okay so it is absolutely Black Box okay so let me now uh in the next uh in

the second half of the talk dive deeper into effectively the protocols that make the fast path and then how we actually you know why we need the the consensus path and how we actually combine them together first I need to to talk to you a little bit about the kind of objects and the kind of transactions that are system deals with because we have structured them in a way to support effectively our distributed system so as you know blockchains we have user transactions and these need to be executed in order to evolve the state of the chain in in our system our user transactions

take explicit s in the form of objects so we have an object-centric programming model where a transaction denotes exactly which objects it will touch okay and the object is in particular reference by an object ID and a version number okay and objects can be one of two types they can either be owned by a particular address and these we call owned object or single loan owner objects or they can be shared which means that anyone can operate on them okay so an owned object you can

think of it as an object you own and no one else can actually change right so an account for example in in the world of uh fast pay is a known to object or a utxo entry in the world that is an address actually that is uh protected by an address in the world of Bitcoin is a known object right but you know owned objects can be like you know characters in RPG games magic swords and etc etc et cetera right that assets that you own shared objects on the other side they are the the things that we're more used to if you want in let's say the ethereum world you know a contra a contract address in the ethereum world would be effectively a shared object right because anyone can basically send

messages to it and can be can can access its data in a controlled way okay so we we explicitly uh do you know the input objects we also explicitly say which command we would like to send the object and you know that basically involves the package name of the move package the programming language we use the name of the the command and some arguments that are just primitive types and of course all of this is signed and what is it signed by it is signed basically by the address of all the owned objects that are included in the transaction and here we have a very important restriction which is that a transaction can only contain owned object by one owner okay and that is the

owner that actually needs to sign the transaction I'm simplifying a little bit we have a few extensions but that's basically already gonna carry us quite quite far okay and of course each of these object references refers to a store that is basically the state of of the blockchain at any point and the story is simply a key Value Store from object IDs to version and versions to the owner uh you know if it is an address or if it is shared the move type and the actual data of of the object okay and also we we rely on on a side uh data structure so so this store can actually be only eventually consistent or Loosely

consistent we don't have very strong consistency of this because this can be big and we might want to store it in a big G Value Store uh one day then we also have the uh object locks as we call it the owned object logs that map for each object ID and version whether that object is already assigned to a transaction or or to no transaction okay and we will see how this is actually used and that store has to have at least one strongly consistent operation we should be able to basically atomically check if it is none and updated with a transaction okay and this that should only happen once the second time it should actually reject it if it is not none and so in that previous figure both paths will be

accessing both paths will access that yeah both paths will access that and we'll talk about this in a second okay well one clarification so so ownership so it's like literally either one or Infinity as far as the number of owners literally one or Infinity in this simplified model in the real world the real world is is much more interesting than this so in the real world we allow multi-sig for addresses natively right so you can actually say an address is actually uh two out of these three or you know five out of these 11 or whatever uh and we also have immutable objects that anyone can access so it's a little bit more subtle in in reality but this

simplified model will illustrate the core of the protocols and the other ones are quite simple to to extend to yeah but in particular we don't have a club good in the sense of only those 10 with that coordination with each other can go for it that's an interesting thought how to do that yeah on the object blocks is it a local view of which validator yeah absolutely so each validator has a local database for the object store and a local database for for the uh owned object locks I'm not talking about global databases here we are building the global database so

we cannot assume there is a global consistent database yeah all right so let's actually uh now dive into the fast path right and which is actually the the path that all transactions go through all transactions in our system first go through the the the kind of like uh check the transaction and create a certificate and then check the certificate before they actually get quickly executed or they go through consensus to be executed so let's actually see what happens here so as I mentioned a user builds one of these transactions that uh that that I showed you earlier and sends it to all validators in parallel and each of the validator checks some local things local

to the transaction as well as uh this uh kind of check on the logs database that is shared amongst all transactions so what are the local things that it checks first of all you checks if the signature is correct if it isn't correct there is no point in actually looking at it if the objects exist if the owner matches actually the owner of the objects it also checks some other things like I mean does it parse correctly is the package available like you know basic checks there so these ensure that it comes from the correct owner all the objects that are referenced actually do belong to that

owner or are shared that's really what what this checks and then the check that really gives you the consistent broadcast is this test and set that is atomic on each of the objects so for each of the owned objects we're going to now check if they already have been assigned to a transaction or otherwise we will assign these transactions to be associated with these objects okay and if indeed this transaction both passes the local checks and also is the transaction that is assigned to all the owned objects that it referenced then we consider that it is a valid transaction and we actually sign it

and send back the signature to the user and once the user collects two-thirds of these signatures by stake for all the validators right for for two-thirds of the validators by stake I should say then that forms a certificate okay and then once the user has that certificate they can send the certificate back to all the validators and now what happens here the validator checks that it is a valid certificate that indeed it is a transaction designed by over two-thirds take all the local checks are rechecked again you don't need to in in some models but we do and then there is a choice here

right the choice is the following if that transaction only refers to Owned objects so the only object that basically contains are fully owned by that address it can send that transaction to be executed immediately through the fast path okay there can never be a conflicting transaction if on the other side it um it also contains for example a shared object not for example if it contains a shared object it has to wait to execute it and in any case it sends the certificate to the sequencing engine or consensus algorithm for sequencing okay and then eventually either through the fast path

or the normal path effects will of the execution will appear they will be signed and will be sent to the user and once two-thirds of the effects are available to the user we basically have our strongest form of finality settlement finality so what properties do we get already just from this part right so if we make the usual assumptions that we make in distributed systems namely that less than two-thirds take is Byzantine or at most the third stake is Byzantine the network is fully asynchronous and cryptography works we don't we don't assume that adverses can

break our signature schemes Etc then if we see a certificate if a certificate exists right it means that no other certificate can possibly exist that refers to the same object at the same version okay this is actually a very strong property if you think about it because it means that the world of transactions just raw transactions with signatures can be quite inconsistent like a a bad sender right can say I would like to send my object to you or use my object in particular way and then also send my object to you right it can equivocate so that's an inconsistent world but the world of certificates is consistent with respect to the owned objects okay

and and that is already guaranteed by the existence of a of a of a certificate okay um and and furthermore there is a liveness property which is that if a certificate exists then there is some owner's validators that have all the state necessary in order to be able to move on and execute that certificate okay so that makes sure that the some good validators in the system will actually have enough data to move the state forward okay and that's all due to the checks that we do and this kind of atomic uh check-in set for the first uh transaction we see per object okay so so that's already a a good set of

properties here and this is exactly what allows us to move and execute certificates that are only on owned objects because now since we are consistent with respect to Owned objects just by having a certificate we're ready to execute and life is good right so let's talk a little bit about the two types of finality that I keep talking about because now you know since we're going to start moving towards uh shared objects uh they will become slightly different from each other so as I mentioned when a validator receives a certificate and does the checks on the certificate it immediately acknowledges that the

certificate is valid to the user and when the user receives two-thirds of these acknowledgments we basically say that the transaction has transaction finality what is the transaction finality mean it means that effectively from now on this transaction will execute come hell or high water it will basically execute there is nothing you can do about it okay you cannot revoke it you cannot cancel it you cannot take it back it's going to happen okay this is pretty cool you can turn off your computer go and have a coffee it's gonna happen Okay no matter what you do okay but you do not know what the effects of that transaction are going to be yet maybe it

will execute and it will basically uh hit a condition that says abort right so so the transaction executes but the actual semantics of the the call may not execute like if what you want is an actual transfer to happen um it may not actually happen now it turns out for owned objects you can fully simulate what will happen so if you have scent a transaction with just owned objects and you basically have simulated what's going to happen and indeed the the transfer or whatever semantic thing you care about uh was checked locally that it would happen you also are guaranteed that it will execute and this will happen Okay so that's

actually pretty cool here so it's the validated it's in a position to know that no the user is in a position to simulate all transactions that are just on owned objects if you think about it right you're the only one who can operate on your own objects your local history so you exactly you you you retrieve your own objects you run the transaction as the value data would run it locally on these objects you will come up to some conclusion about what the result of that execution is going to be well guess what the validator is also going to reach exactly the same conclusion so for owned objects that finality is very strong and in fact since we then can take the fast path and execute immediately the second form of

finality will come literally milliseconds after that finality so it's not particularly you know different but then after execution right either a few milliseconds later if we're just talking about owned objects or a bit later as we will see for shared objects we also have a second notion of finality which is settlement finality this means that effectively in a quorum of validators the goods have exchanged hands right so what it means that from now on you can make transactions on the basis of the settlement of the previous transaction so for example if Alice has sent some asset to Bob Bob can now at the point of settlement finality put in a transaction using that asset that will

succeed okay so that's the settlement finality yes so I presume there is some stake involved with the validators that's right so what happens let's say I make a transaction I receive two-thirds votes with a third set of validators and let's say a year from now the set of validators have all changed yep to a new set of validators and at that point you know there's no notion of time over here there's no notion of ordering that's right the new set of validators can say okay right now I'm paying to George but tomorrow I can go in and paste the same money to Tim and a new set of validators are voting for me no that's a very good question so our system is designed to operate within epochs within an Epoch the validator set and its stake is stable

and everything that I'm saying here is with respect to that stable set across the e-box we have um effectively a reconfiguration protocol where the the stakes can change Etc and I will make the argument that all the safety is preserved by that protocol so your right to worry about it but I will tell you why you shouldn't worry about it and the reason you shouldn't worry about it does not come down to certificates because that not not to this particular part yeah there is some notion of consensus over there when you move from one Services you have to you have to have consensus

to do that yeah there is nothing magical we haven't like yeah we haven't uh uh we haven't had a miracle in being able to do that without consensus shall we say yeah yeah so you can think if you're one of a certificate that's being valued within one Epoch that's I think the way to think about it and everything that I'm saying before I talk about reconfiguration is really with respect to that one Epoch yeah okay cool what is it back and uh one day one day one day I mean it's up to we could change it but one day it seem to be seemed to be the natural uh unit for a knee pocket it's stable enough but not too stable yeah

all right so we will we will talk again about this settlement finality when it comes to Shared objects because in the case of shared object it does come a little bit later um than this but what I wanted to point out and and Karthik to some extent alluded to this one of the the challenges here is that once basically a transaction is fine or even after two round trips here right in in the acknowledgment we need to guarantee that it survives across epochs and that it is included in checkpoints otherwise we we cannot actually get safety in the long term right and and this despite the fact that

things are happening asynchronously in everything that I have shown so far okay so so that's going to be an interesting challenge now let's talk a little bit about execution now one thing that you may have noticed already when I presented the fast path is that both the check and of the transaction and the check of the certificate largely look at things that are local to the certificate and largely look at a very small amount of state that is related to these objects that are included in the transaction that means that we can basically Shard both the check of the transaction and the check of the certificate and the only thing that we need to be very careful when we we actually Shard that across different cores or different

machines down the line right now we do it across course is that strongly consistent locks table that needs to basically record the first transaction that touches an object but not the second one or the third one okay so there is a very small critical region there where we have to make sure we have synchronization between different asynchronous processes that that do the checks but everything else basically can otherwise parallelize across all our cores and we do that okay now when it comes to the execution for um for uh transactions what we basically get is we get the transaction certificate and we also query the database for these particular objects

that are included in the the transaction and then we just run the move command and what comes out is a set of effects which include which objects were mutated and at what version which observes were created and at what uh um version deleted as well as some status and some events that that they Emit and and what we basically do once we base once we we we conclude the execution is then we go for the new objects or the mutated objects and create entries in our log tables for the new versions of these objects or the new objects that were created okay so as you process effectively transactions you invalidate or set the logs to the transactions and

create new logs to actually continue processing the objects as they evolve in the system and we use Lamport time stamps here to make sure that object IDs and versions are always fresh you can never actually go back to any previous uh to any previous one so so there is a a sense by which we always make progress forward which allows us to make quite a few optimizations in not keeping state to check that things are fresh to avoid replays Etc now one thing to notice here is that the the processing of transaction again only touches a very small amount of state and again we parallelize that across all the cores in our system and we have designs to actually distribute

that again across different hosts uh on the validator on the basis that we know exactly what state it touches we can efficiently retrieve it to the computation and you know restore the new objects that are created okay so shared objects let's talk about shared objects because this is really where where the the challenge is here the the remaining challenge what is the problem with shared objects why can't we use what I described so far to process them the problem is that because shared object could be included in anyone's transaction it is difficult for these disparate parties to coordinate and propose a

consistent sequence of transactions with a consistent sequence of version numbers that these transactions should touch okay so the system has to come up by itself with a sequence of transactions and the versions on which this transactions should operate on the shared objects okay that's a harder problem the system must assign the version numbers and and that sadly requires consensus right and that has many more critical paths effectively uh within the validator where we need to do synchronization Etc okay uh so so this is how we do it right this is how we do it on that side on the left hand side we

have the fast path and eventually a transaction comes in to consensus uh with a certificate and the certificates are sequenced now what happens here is all the certificates are sequenced all the certificates are eventually passed to the checkpoints okay but if the certificate contains a shared object what we do is we assign in sequence the next uh version number for the shared object that that it is touching and we can do this statically we don't need to execute at that point to do this and we can do this statically because we maintain a a database of object ID and what is the next version since the versions are determined by lamp or timestamps we don't need to execute uh

these uh these uh these transactions to work out what the new version is going to be we just look at the input sequence numbers we take the maximum we add one that's basically going to be the output sequence number so all of this is extremely fast it literally involves you know some computations on integers because that cannot be parallelized that is the sequential aspect of our system okay but guess what modern computers are extremely good at doing that and then once basically we have assigned a version we just basically store a database of okay this transaction you know this transaction touches basically these particular sets of shared objects at this particular sequence numbers and now what we do is we inject those certificates into our execution engines and our execution

engine takes uh the certificate looks at the transaction for the objects that are owned the version number that they should access is already given by the creator of the transaction for the shared objects the version number has been assigned after consensus and we're golden right we know exactly what state to execute on there is no room for inconsistency between validators or validators will execute on the same input objects yeah you run the risk of any collisions like if for the objects that are owned objects could have a user pick the next version number in a way that it like collides with one of these shared objects yeah yeah so we have to be very careful about how we derive our object IDs and the

versions in order to make sure that we never can recreate no user can ever recreate the same object ID version okay so the object IDs are actually cryptographically derived in the transaction by hashing all the inputs the transaction and that lead to basically the sequence of new object IDs that are allowed to be created so that cannot go back there cannot be Cycles in hash functions so they're we're safe and the version numbers are created using these Lampard timestamps and we hope that the since we're using u64s for the version the number of transactions you have to do to actually make a lan Port timestamp roll around is so large that we will be very rich uh before it ever

rolls back so it's someone else's problem I mean we did back of the envelope calculations of how many transactions you would have to do we're talking about you know thousands of years so we're not worried about it right now that it will roll around the user has no uh discretion in setting version numbers the system always sets what the next version number is is there no notion of priority the user can't presumably pay more to that's a very good question right so so this really depends on how that component that those consensus really is is designed to us it's a black box right so so there is a sequencing engine underneath that gives you a sequence what really matters I think lots of people think of consensus

as a sequence uh in in my view this is wrong and I'll give you a bit of an Insight here to me a consensus is a distributed protocol that gives you a sequence of sets of transactions then the system and I'm talking now about the system around the consensus can actually apply its own logic if you want for how to sequence deterministically that set within each of the commits so there you can order by gas fee for example which is actually what we do right so we take basically um you know the the kind of the kind of consistent execution we order within that by gas fee and this way we we prioritize if you want the less uh sorry

the the more expensive transactions first yeah but there might be other approaches there I I'm not claiming this is the best approach yeah yeah so you're saying that in your consensus protocol it actually does not have discretion over the sequencing like it's literally just in some ways that's just like automatically interpreted the execution layer or what the corresponding sequence is so what I'm saying is that one would look at basically the the sequence produced okay let me back off here like let me let me take a step back so traditionally what State machine replication needs is

a sequence that sequence doesn't have like natural boundaries of commit sets right that is a problem right because you don't quite know what the latency characteristics are it might be that you get 10 items and then the 11th is after two seconds of late and zero it might be the case that they're all in the same commit set so the 11th is actually already there right this is not a very helpful abstraction for a system designer that actually cares about performance so the way I see it is instead a consensus algorithm provides a sequence of sets of transactions and within each set you have the same latency effectively and now the the higher level system is free to reorder those because reordering is free within that set right as long as you do it deterministically so one way

of reordering is to say let's take basically um you know the the most expensive first right and that's for free and you know you you that's one way of doing it you're really seeing this sort of two different sequences you care about one is this kind of like wall clock time and the other is kind of actual execution in the virtual machine right that's right as you're saying the consensus is sort of like more concerned with the wall clock time transactions in a single block are kind of tied with respect to that that's right and then the ties are broken you can break the ties in many ways we do it in the simplest possible way yeah oh good yep so it's up to each Bell either to decide how they do the sequencing within sets or is that specified by the overall protocol it has

to be specified by the overall protocol right because it has to be deterministic eventually when we reach the assignment of uh transactions to Shared object versions all the validators have to agree so to some extent the way you break the ties and the way you actually eventually reach a sequence between shared object has to be agreed amongst all validators that's the state machine replication Paradigm so the discretion is the system designers discretion on how you take those commit sets and come up with a sequence but everybody has to apply that yeah yep cool all right so this is the shared object path um now besides assigning shared object locks which we do in order to do execution on

shared objects consensus is absolutely necessary for checkpoints and reconfiguration for checkpoints what we do is we take all the certificates we stick them basically in a block but because things are asynchronous it might be the case that some certificates that get into the block depend on other certificates that exist will eventually be processed but are not yet available okay so actually checkpoints have to ensure and and we do ensure that that there is actually a a a a complete sequence of execution and we also ensure that the checkpoint order certificates in such a way that if you were to execute the most sequentially

with many of our full nodes do because they're not super sophisticated about dealing with these things we want to make it simpler for them right you actually end up in the in the the correct state of the database and since since things happen is synchronously both in the the consensusless path and also in the execution path this is not guaranteed actually so we actually have to do some more work there to make sure that the checkpoints are kind of nice and clean as if it was a traditional blockchain because lots of people want to consume that so we need some theorems here that are not trivial such as if a certificate is sequenced we need to ensure that eventually all the certificate that it depends on are also sequenced otherwise we would never be able to make checkpoints right and we

also need to ensure that eventually all final transactions even if they're final through the fast paths that is not relating to the uh to the consensus if you remember do make their way into checkpoints right so again this is not a trivial property if you have fast paths cool okay uh I I seem to have missed a slide I apologize I see checkpoints that should be a slide on reconfiguration I apologize my I copied the title uh so that's the final component of the system that I want to talk about it's not checkpoints it's reconfiguration so finally

we want one last thing from the whole of our blockchain which is we want to be able to change the set of the validators periodically once a day okay we call that period an Epoch within the epoch we have the same set of validators with the same stake across epochs we want to be able to change it okay now what we do uh for this is that eventually validators stop signing new transactions that means that eventually new certificates stop being formed and then what we need them to make sure is that when they basically see all the certificates that they have received being sequenced and being executed they basically say I'm ready to close the epoch and we have a theorem that says

that once we have two-thirds of validators by stake that vote that they want to actually close the epoch we can close the epoch and all things that were final in the epoch will eventually make it into the sequence will be in checkpoints and therefore when we open the new Epoch from the database that results from these checkpoints and that sequence all the state will carry through and that gives us safety across uh across the ebooks for reconfiguration okay and this is basically an overview of all the components of the system and how they interact with with each other so just on that last point so the if you have a certificate for an epic it's guaranteed to be in the checkpoint

that concludes that is so potentially carry forward to the next that's correct so the guarantee is if I get finality for a transaction within an Epoch I am guaranteed that the transaction will appear in a checkpoint within that ebook and that's it basically yeah because if it didn't it wouldn't carry over to the next one right and therefore it's not finality yeah yeah so I guess like for that last point about liveness is like the concern that like if a transaction grabs a lock then like builds and doesn't like properly

move the object to some other like or something then you can't actually touch that excellent excellent question thank you for your question because I was about to move forward and forget about it so upon change of epoch something else that is very important happens which is we reset all our locks in the owned object locks database so that means that if your client is buggy you have an object at a particular version and you create two conflicting transactions none of which manage to get a certificate within the epoch your object is locked right in fast pay your account will be locked forever but here because we have as a point of synchronization the end of the epoch right where everybody is guaranteed to have exactly the same state everybody

being the validators we can basically pop and delete all of these logs and start anew for the next Epoch which means that a bug will cost you a day it's not great you're not gonna throw a party about it but it's not going to cost you a fortune forever which is something that was actually concerning is that answering your question yeah yeah thank you very much yeah that's actually a very important thing from the standpoint of a client how should I think I mean in the same Epoch I think it's easy I see a two-thirds certificate I'm done right but today I guess when when you order everything then I think the way I see a transaction is if it's a single owner transaction single owner

object then I see that the transaction happened you paid me money and then there are no transactions uh so so when you're paying me money I'm just trying to figure out if you already own it and that is you know you haven't paid it to someone else yeah yeah yeah so that is what you check for but in this world you you have this two-thirds certificate but if once you have you know reconfigured multiple times do as a client do I need to know who the right set of validators were at that time so that I can check hey the two-thirds are actually the two-thirds set of violators no no no no no no no no not at all not at all so within the epoch if I'm the creator of a transaction

within the ebook either I see two-thirds of Acts from validators or two-thirds of effects and then I'm convinced that it's either gonna happen for sure within the epoch or it has already happened within the epoch now good within the epoch now within the epocket can also transfer the certificate on the effects the second point of finality to anyone else and they can convince within the epoch that the thing is done so you know if I want to pay you to buy some shoes or something right uh I give you basically that certificate you're like okay great I know this thing has happened even without looking at the chain I'm good it has clicked on my account I give you you give me the shoes that's it right now across a box right we don't rely on cryptographic evidence of

certificates from previous Epoch what you do is you have to to follow basically the the reconfiguration of the committee going forward so when we reconfigure the old Community science the new committee so you you have effectively a chain of new committees you make sure that you're in the last committee the current Committee of the the current day and then you go and ask basically you're like get object right literally get object at that version or get object by owner and you have two-thirds of the the validators telling you oh yeah this object ID comma version belongs to you and then you know it belongs to you you don't have to go back and and ask

or if you want to to like be super certain like which is an auditing function it's a full node function you literally take that last uh certificate sorry this last committee uh but you also take the full sequence of four commands in the system and you can re re-execute them right I mean but that's like a full loaded of a full node yeah cool all right just to to say yeah we implemented all this uh the whole Suite is a big system it's 250 000 lines of code the actual mechanisms that are designed that I described is only about 30 000 lines of code it's not like huge actually now that we know what what they should look like we should be able to

like have them being like a kind of like summer project for like uh you know a student or something but you know when you don't know what you're designing it always takes you a long time and a lot of lines of code to actually work it out it's about eight thousand four hundred commits uh and about 70 people worked on all the different aspects of uh of the system including all the researchers that actually designed the system so all of our researchers basically we're we're designing and coding and and doing all the kind of QA activity so everybody was was involved we had three test Nets and as part of our test Nets just the last test net in fact in April we we basically threw millions and millions of transactions and objects at this system

just to to work out all the Kinks and um you know debug it and we induced very large peaks of 130 to 200 000 um transactions in transaction blocks I haven't talked about transaction blocks peaks of about 10 000 transactions a single transactions single certificates as I described here and since uh 62 days now or you know maybe like 50 days more likely we have a main net and we also have about 12 million transactions and about 10 million objects uh in in the system so it's you know it's doing the job that it's meant to be doing and all of that software is built so far to to do parallel uh parallel stuff both the

execution and the checks and all that stuff but within one machine on on different cores right we light up all our cores and of course we are waiting for the day uh this supply of transactions exceeds uh you know what we can the demand for transaction exceeds what we can supply to actually move to many hosts it's uh it's we're not quite there yet okay now there are many aspects of the system I haven't talked about I just want to call that out don't assume that because I haven't mentioned something or because I have simplified something we don't do it we have many other features one feature that I think would be interesting if if people want to chat about it uh over launch or something is programmable transaction blocks I have described how you can parallelize operations on different

objects so if you own 100 objects you can do transactions in parallel and all these hundred objects you understand that you know this is the topic of the whole talk but what if you want to do transactions that really depend on each other you want to do a hundred transactions all have dependencies on each other in theory you would take delays of a few seconds every so often right to transmit the transaction wait for certificates maybe consensus and then the next one cetera that will take you minutes so we actually offer A Primitive called programmable transaction blocks that allow you to literally put them all in one block and run the whole thing through which is a very powerful kind of like scripting primitive for for blockchain so I'm happy to chat with you about it and anything else here

and in terms of performance what we actually have at the end of all this is less than 500 milliseconds for owned object transactions so this is quite remarkable we can basically do smart contracts on objects that you own yourself for less than a second payments uh you know Inventory management transfers and all of that uh for for very uh at very low latencies we'll actually observe like hundreds of uh transactions per second if you put many of these owned objects in blocks as well or about ten thousand if you just have one transaction per per block uh in in the in the chain and for shared object transactions we basically largely

inherit the performance of the consensus so we observe about um two to seven seconds of latency you still get transaction finality after like less than a second but then you have to wait for the actual execution to happen so that's an interesting thought there you can basically turn off your computer and go away in less than a second but it's going to take a few seconds for for the effects to actually manifest and our our performance again for shared objects is is roughly the same it's about 7K TPS for single transaction transaction blocks and it goes up basically as we have more more of them

and that's it uh in conclusion uh the suite Foundation actually has research grants some of you may just be interns uh and and your research groups uh might be interested in funding so uh do actually send us a short proposal if you're looking for uh to do projects that relate to the three ecosystem would be very happy to support you and in conclusion production systems basically are never neat this is one thing that that I want to to Really highlight from this like am I satisfied at a kind of like you know abstract level about how the system looks like namely it's this kind of you know mishmash of two different systems and and all the complexities in order to

actually try to get the best of both worlds no did I did I you know become a scientist in order to to design such complex and and ugly kind of like mishmashes or systems no right but actually in reality people want this like you know this is what real systems look like because some people want to do really low latency transactions and some people want to do transactions that look like a Dax and you know they also want to mix them so a model like Fast pay where you have a side chain and then you bring them back is not good enough right so real systems are messy and in particular when you take uh real systems and you have to add features robustness scaling and

performance they become even more messy but at the same time you know dealing with that messiness without compromising on safety and also performance is a fascinating area of research so I actually have no regrets that despite the fact that my intellectual Purity would not find appeal in in this kind of systems I'm actually very uh glad that I'm working on them because the the the wealth of problems that you find is unmatched by uh you know the problems you try to solve when you have a more clean theoretical model of what you're trying to do in your system so I I encourage everyone to to look at sui and other systems and actually engage with that messiness philosophical question but when you said

people want this who are the people how did you determine when building a system right that obviously didn't have any users at the time you were building it what people actually want like the the complex design of ethereum can sort of be explained by they built a relatively simple thing and they've kept adding stuff to it based on how people were using it whereas your more right building a whole system from yeah yeah no that's uh that's a good point um so it is actually a thesis of ours that effectively there will be demand for blockchains right ultimately I think it's a thesis that we all share here uh

that to some extent if blockchains are useful they will be popular in terms of the the volume of transactions that you will try to do and it is to some extent you know a further thesis that the the simplest transactions actually take a lot of the block space like transfers actually are quite prevalent in blockchains I mean this is how we basically came to that model we looked at other blockchains and we're like what are the common types of transactions and really there are two types of transactions that are common there is the kind of simple transfers be in ethereum erc20s or native or whatever and Bitcoin is all transfer so there you go you have a simple you know chain that

is all about transfers for a while cardano and others were also largely just transfers and then you have these other transactions usually dexes um that you know are basically kind of shared object transactions and these are the two kind of things that that happen in the systems so we're like if we can basically really nail the latency of you know the the transactions where we can the transfers at least and expand a little bit the reach of that model then we have an edge right this is where how we came to this and the second way we came to this was literally I mean for the fast pay uh work in particular

that was the vision of uh Facebook Novi that they're going to do just transfers that was all that was all basically DM was doing at the time at a phenomenal scale at the billions of users scale now this didn't play out right but that was the the client asked yep uh can you speak a little bit about the requirements on the validator can you run a validator of a laptop um no you cannot run your validator from a laptop unless you have one of these Mac m2s at which point you should run your validator from your laptop right um no what I mean by that is we never really designed the system

to run the validator from a laptop um if you try if your system does a transaction rate like a few transactions per second or 10 or 20s transactions per second I'm talking about you know Bitcoin ethereum kind of levels of volumes then you can run it from your laptop it's not a problem at all right you need a modern laptop but like you wouldn't have zero problems doing that if your system does the thousands to ten thousands certificates per second or hundreds of thousands of transactions within those certificates if you want then you do need to have 24 cores and you need to have I mean it's not just a compute that

is the problem you need to have one of these uh um nve disks that you know offers very low latency and you need to have enough bandwidth to to replicate that volume right so that's the the place where I'm like we're not we're not designing the system to run on a laptop because once it graduates from being you know a small capacity system you have to process that data it's even full nodes who yeah one more question about when you when you're saying the user collects certificates from the validators how does it practically work oh yeah very

good question yeah yeah so when I present the protocol I present it abstractly right there is Alice uh she sent certificates to everybody she got a certificate in practice what happens is that our our full nodes effectively act as a Gateway as well not hours any full node acts as a Gateway so as a user you pick a full node that you usually do business with or one or two or whatever and you send your certificate your transaction there the full node basically deals with a protocol because once the transaction is signed the rest of the the the processes doesn't require any secrets anymore from the user so it can be driven independently of the user so um we we delegate that to full nodes that go do all the communications gather certificates and eventually give them

back to uh to users and the user doesn't have to worry about that yeah let's put the validator set so is there is there like a bound on how big that can be or is there a minimum sort of stake amount or so we do have a minimum stake amount it's about 0.25 of the stake so that means that the theoretically largest possible set we can have is 400 right right now we're operating at a bit over a hundred um I have to say the protocol is not overly sensitive to the to the number of validators uh so it extends to hundreds I think extending to thousands is really about networking efficiency Etc you you

would have to start doing different things at a networking layer to to make sure you can basically talk to each other but even though from a theoretical point of view there is a kind of n-square per validator cost that goes up as the the set end grows or and Cube I guess for the whole system the constant in front of that is so small and and in comparison with actually shifting data around um that we don't really see it per se on on the set uh one one thing that we observed that was actually much more worrying initially was the fact that since you do full replication you need to replicate to all the the the full

nodes for example so initially we we didn't optimize these kind of read paths um and that actually led to more worries until you know we actually created distributed ways of sharing all that data in a kind of more hierarchical way and now that that traffic never touches really on validators yeah you differentiate between the owned object and shared object is it like a land move language constraint we have a flag in the database it's uh yeah so so literally um

are our database has a record for an object in a particular version and uh in there you have an owner field that can either be owned by a particular owner or shared and that's it so the user when it creates an object that identifies which type it is no it's the contract right so so all objects are mediated and created by contracts they are proper move types and all that stuff so the the move contract creates by default owned objects and then you have a kind of like make shared function that lifts it into being a shared object and that sets the flag and there is no no going back there's no going back you can delete the object and create another one if you want a known the object but the same object cannot go back into being on all right thank you

so much and thanks for your questions