Debugging the Consensus

Transcript

okay next up we have a wonderful core developer from Paradis who's gonna be talking about debugging the consensus which sounds very important if I do say so myself so let's give a warm welcome to way Tom hi everyone so so I want to talk about our consensus failures and what I called reliability engineer for a serum so if you remember our last Hart Forks Casino Pope unfortunately things really didn't

go really well so it cut delayed twice we have to implementation bugs the first one is related to 1283 which unfortunately caused the first ralston consensus speed and caused the first delay and fortunately the second one 10:52 didn't get exploited we also have a subscription pack also related to 1283 that is cause that can result in some we infuse for some contracts and it's called a second delay so this

fortunately nothing goes into the net that's a good thing but still we we we still can learn a lot from this process and even seems only having enticed not even it seems only happening security audit we still want to make sure that they works reliably so a natural question to us is how can we make the issuer network more reliable from our experience learned from our heart past heart works talking about reliability easier there is one important concept that you already caught first describes that is acceptable risk it basically

stays at hand percent reliability is not only impossible is also worse for most our users so the hard truth is that our developers are humans we are just humans and we write bugs as the hard truth so the issue is also that compared with a high reliability system and a really high gravy system or the user may not really notice the difference because we the transaction only get confirmed in sorrow second and we also have the most miners exchange only expired a transaction to be confirmed at least ten

minutes or half an hour later and we're not talking rocket science meaning we're developing a living system this is a system we can definitely change after it applied so it's definitely really good if we try to be really careful before deploy anything but if something goes wrong it's not an octal board also for blockchain field are really important seeing is that we need counsel innovation and if we side apart too high for reliability the issue is it may prevent us from adding new features to the blockchain and that may be a really

bad thing for us because it means is your may be outdated so we are decentralized system and that means we are talking about two different reliabilities the first is software reliability that is just related to the no dimple implementation whether the no real crash whether the node just behave as expected I know I also have two network reliability which is a slightly different concept this is related to the overall network health I put it in as a different liability because you know dissimilar system yours had easier in the case that even if our individual

software has some bug and it crash but the overall network may not really be affected so for software reliability we have improved our process a lot we have done a lot more code review we have done security audit for our client and also for publications we write we also have an integration test that runs for all the the node implementations and in the mean time we also do a lot through the testing to just give give out procedural random of code into different client and make sure

they get the same results but for network reliability that is what actually you really care about so we have we need to ensure the state safety and we need to make sure the network is continuing running it must confirm transactions you must produce blocks so we really focused a lot on prevention that is most for software reliability and network reliability but our past experience shows that we really need also need to focus more on reaction which is if something bad happens what

can we really do and I even want to take this one step further that we have post software reliability and network ability and we even need to try to use some architectural solutions to try to decore relate those two different liabilities if my software crash here is the back how do we make sure that network is affected as little as possible ok so so so this is anyway

so okay so so if you remember casino host the first thing that caused the first delay is because of consensus split and I important issue is we can better communicate consensus bug by providing some subcategories so for example if a note sorry anyway sorry so if so for example the we may have some consensus bug that can cause a node

to crash and in this case it may not really affect the network so or we had some have some denial of service attack on some substitutions this is your lira lates you to some guys calls being said to lope and in this case we can this this case can affect the networks through food because we may have some spam transactions that cause normal transaction not being confirmed all the most dangerous one is actually conscious a split which I will explain why is so dangerous and we probably also had some

structural failures like if our capital algorithm is broken or our proof-of-work algorithm is broken that is basically tombs day but it never happens you only witches organized those just as critical but it really benefits from some subcategories so that users can better understand the the difference of the severity and from our past experience most of our consensus parks happen between the eevn interaction between the EVM and blockchain state and also unfortunately the consensus split is

also one of the most common type of consensus bugs if you look into the history the the issue is also that currently this is also one of the most difficult to debug it can take a lot of developer time and for consensus split we do we already do a lot of prevention that really help a lot for examples or integration tests and also for the testing and here I want to talk about this related to reaction so the issue is most consensus Park is

surely permanent like like when it happens and to fix is released it really cannot share self queue so we only ask miners as his friend to run multiple implementations to run multiple nodes the issue is for the case of consensus split running multiple nodes work I only help detecting at least until right now and this means once the currents - please - it really is a rush of time so developers must even if happens in midnight all the developers must kind of

ride and try to fix the issue immediately and try to release the back and that is really not cool experience for all of us at least working to try to fix the issue also probably the most important thing for users is that the blockchain is Karva bulletin for at least several hundred blocks because because this please is permanent we only need one or some subjective peek of what is actually the conical chain and that can take sorrow at least several hours until there's no miner left

the other side which is pretty bad first considered we need to consider the whole blocks hanging style for several hours so why why this is the case so if we have some blocks and so the kind of split you see early - when we have some you really like this so we have some valid block first and and after that the node it's two different nodes probably not a January's blocks and only no they will accept and no Bissell is invalid block also note B will generate a block

that note B only or not be exact but no result that is only valid blocks so in this case no matter what is the hash free distribution from this point no they will never react to node peace block vice versa so in this case no they will only build on top of no day's block and no P will only build on top of no peace block so in this case unless no they will not be release of fix it will continue to be to have multiple conical chance which means no chemical chance but the thing is this is Tinder's but we can

actually make it safer so we can make consensus pleat self-heal by trying to find a common subset of transactions including a block or settled by no nodes and we make the majority if we can make the majority manager Chan know even in the case one cannot simply split happens we still have a conical chain so this is basically like this if we can try to find a common subset of collections are still excited by all blocks then we might have some minors major vlogs that only know they're yourself

some manners may generate some blogs that only not be accept but if it has a common subset then both no day and nobody will be able to react into the comment subset so it's a common subset really have the majority of hash power so we still have a conical chant so how do we do this so this is a already will establish tech techniques called immersion programming so basically what we really need to do is we need to split our node implementation into some modular components at least into two

first we have an importer which handles importing blocks and try to validate all the consensus rules and we have a producer which handles transaction pools and also for those new blocks so if we make the producer validate transactions against beautiful UVM implementations and for the resulting block it only includes those transactions that all the usual I'm from implementations of Greece so we find the common subset so this is immersion programming it's say for now so we always have a conical change the network's always shaft and we don't need

to get up at midnight to try to fix the back but of course but our true be fixed as soon as possible we still pay something because it's to a consensus split in this case the producer will be entered in l service attack because our tracker can kind January transactions that he or she can be really sure that will not be included in any blocks unless the the consensus bleach bug is fixed still this is arguably safer compare ways of who we

have mutiple chance and who network is that and a second thing we need to pay is that currently most Noli implementation are not really designed for this we don't really have a modular design right now so so using a version programming will almost require all know the implementations to do some refactoring if you remember the second issue that caused Cassino foe failure delay that is due to a backward ability issue in 1283 and we really learned a

hard lesson that backward habilities is complicated so after the the reentry bug was was discovered there was a lot of discussion related to embarrass which is doing a hard fork what are the poverties that we should preserve and what are the properties that we should change so what is the internal behavior of the uvm what is now the internal behavior and what we really should make immutable for the blockchain the issue is we always have developers who might not follow this rule we might

have some developers that just want to use some edge cases which probably save some gas for their smart contracts but might be broken you know your filter hard work or we have we can some have some developers that use antipodal codes and in this case the order is probably something who can if we deploy deploy something that we envision in the future this OP code will be able always kind of have developers who just want to assert certain and guys codes for

for some alterations or in the really bad case which we are still telling with that in some cases the data is pushed at code onto the blockchain which actually caused a lot of troubles and the issue is that it really doesn't matter what environments you preserve because it just because there will be developers who don't use those environs and well the number will be really low when the

assume community continue to grow we will it will be the case that even a really small fraction can be a really relatively large number and the issue is that what do you preserve the safety of those people's money for any hard work so this leads to what caused the whole room slope which is love from a lot lot of observations when designing a system so it basically says this with a sufficient number of users of an API it does not matter what you promised in the

contract o observable behaviors of your system will be dependent on by somebody so so so the issue is really that even if you're having environments we can still have contracts that use something outside of those environments and in this case what what you what can we do what should we do next time to a while the same issue by this time we cut for the colonnades the 1283 reentry back so probably let's just fixing the

software engineering way so we have something called a conversion which basically allows contracts to stay is current state with the exact gas hose with exact of course enabled there are few designs already which some of them try to just add a new at a new are LP item into the account state some adder will first need to disallow deployment of a new sub course and then we can just simply add a prefix for any account code

or we can do some clever trick and just store the account code into a into a smart contract so yeah so so I think we really can learn a lot still from our past hardwork issues and what I really want to say here is that ice developers for a serum note we are not only maintaining our software we need to make the software robust we need to make the software to have as little bug as

possible but in the meantime we are also maintaining the network and it is also important that we try to fund some architecture solutions to to try to make the network more reliable even in the case when our software is not actually reliable so yeah that's about real be engineering and I have a longer version of this presentation if you want to take a look and for Part II we are hiring if you want to work for park adult sub strict ECM is zero and two one zero and check this link and that's it thank

you [Applause]