Fantastic metrics and where to find them

Transcript

bad jokes aside so before we even start this talk I'm gonna see what was the motivation behind it so basically for the last year and a half I haven't been working on a project which I'm not still going to mention and I have talked to a lot of dab developers protocol developers and basically a lot of people working in the blockchain ecosystem it started talking about monitoring and again I'm kind of I'm going to come back to that later as well what monitoring actually is but talking to them almost no one either heard of it or their use not even using it now with the expense of the defi ecosystem and stuff like the folk Kermit tack and everything else happening there is a huge huge huge demand for monitoring and metrics so we don't things up and then screw the whole system in user users money so

first of all hello everyone my name is Bogdan I'm one of the developers and a co-founder working on tenderly and tenderly is a blockchain monitoring and alerting platform where you basically import your smart contracts and then you get all of these nice metrics and development tools which we might see you later on anywho so the title of this talk is fantastic metrics and where to find them and usually coming up with the title this good takes more time than actually preparing the whole presentation so today we're going to cover basically three topics the first one is what is monitoring the second one is when to start monitoring and the third one is monitoring taps and protocols themselves so starting with the first one this is like the usual wakeup edia thing that you see so

monitoring is tools and processes but by which you measure and manage your IT systems this basically tells us nothing it's just a really nice x rate you can leap read on Google when you search but how I like to see monitoring is you have some metrics which are generated by your system and by system it can be your DAP it can be your protocol it can be your fronting application or whatever and transferring those metrics in something the business can use so here we can see two things the business and the system now by a race of hands how many of you describe themselves is a business person in your team okay now I'm gonna try and ask everyone to raise your hand and say that the correct answer is everyone here you are a developer but you're not doing

it just for the fun of it you're being paid for actually bringing the business some value so from the business perspective Matt for monitoring our user experience data now what we mean by user experience data but user experience data we we mean stuff like let's say for example I'm Pro proc roof of work which I'm not and if I put it on the top of my website do I have more users coming or not it is used for product decisions so if I have more users coming in does that mean that I should actually invest in my team pushing that forward of course the answer is no but that's another discussion totally and then from a systems perspective and by systems perspective I mean developers designers and everyone else that's actually building out the

product themselves it's the state of the system so if you're making a blockchain game for example this can be number of games that are ongoing or let's say if you're doing like a loans thing number of loans that default it for example or stuff like that the next one is used for detecting problems now a lot of developers actually use monitoring in some way you have some logs that are screaming you receive an email something isn't working and then you go to your logs and see that there's something actually happening and then finally you maybe use those logs you got for a bit that's SH into something or go to ether scan and diagnose and resolve all of the faults that are happening so that what monitoring is but what most people get wrong is when to actually start

monitoring and it's actually really well perceived through the system called the three levels of monitoring so basically we have three levels of monitoring we have the manual one and we'll cover each one of them we have the reactive one and finally we have the proactive one now while I'm describing all three of these keep in mind that around ten percent of the teams use the manual use the manual method about 60 percent of teams use the reactive method and then finally twenty percent of people use the proactive method and then try to find where you actually fall so the manual method is we check things periodically so I don't know if you guys ever did this but for example you wake up open up ether scan and see how many transactions failed for your contract and this oh ok something

isn't working in the last eight hours there was many transactions that fail they shouldn't have and then you need to check it and this is horrendous by the way this is a talk where you should feel guilty while listening to the then the second one is the user lets us know something isn't working and I can explain this one in an anecdote the previous company I worked with had the ticketing system for the helpdesk and after about eight hours people actually started screaming on Twitter that we're not answering their tickets and for the whole day we thought oh my god everything is working everyone's happy nobody's sending in tickets but actually the ticketing system itself wasn't working and then finally it is usually added after a failure so after we discovered that the ticketing system wasn't working then we added like a

check that's actually going to I don't know send an email every one hour and check if that email was received so this is bad the second one which is mostly used is the reactive method so it's based on simple thresholds in normal web 2.0 development it can be for example your CPOE is over 60% or in depth development the one that I mentioned for a failed transaction it can say okay if more than ten transactions fail per hour send me an email and I'll check if something's wrong the second one is there's some movement towards using that to measure customer experience so we know that the customers don't like when something isn't working so we're using it to fix in it that way but also for example you

can say okay I have this new version of the contract in the best way actually to explain it when maker was moving from site who died they could have used the number of migrations from site to die and see if people are actually moving their funds there so that's using the metrics for customer experience then updates tend to be reactive in response to incidents and outages so here we still have a problem that we might put something in place but we're only going to change it if it we see it fails so it's usually one of those things don't touch it if it works and then finally you the last step it's the last step in application development so you don't do it from day one it's you finished up your app you're done coding you even maybe deploy to main it you have new

users and then you think okay something might actually break I might some put put something in place so this is like 60% of teams use this approach and then finally we have the proactive one which is the correct way so it's it's a thing where your metrics are the core of your business but also of your interest structure and that's why I said everybody should arranged your hands when I said business they also focus on measuring application performance and business outcomes so as I mentioned let's say you have a new version of your contract so a business outcome is for all of your users to migrate to that new version so it's not the technical thing it's a social thing it's a business thing so you see that the new contract

is bringing much more value there is a focus of measuring quality of service and customer experience so even while you're developing an app so you're a developer a hardcore developer you use then you write solidity you compile the code yourself you deploy it yourself to web by a web 3 you still need to think about how each line of code affects the customer experience in the quality of service and then finally and this one is the most important products will not be considered feature complete or shouldn't be deployed before metrics are added and this is like the thing that should guilt-trip you the most so without having anything in place and for smart kontrick this is also auditing stuff

this is like external checks you should do all of this before deploying stuff we're spending other people's money not our own and then basically let's go through what monitoring means for the apps and protocols and now because my computer couldn't be connected I'm gonna try doing this one by one and it will probably fail at some point so the first one and the usual one and the easiest one is gas usage so basically because unfortunately PDFs cannot play movies let's hope this works of course here we go so basically gas usage you can take a transaction and see how much gas your transaction uses you can deep dive into

it and actually see which function uses the most gas that's the first thing you can track for example just to discover how you exit this thing sorry here we go okay so next thing is optimization comparison let's say I'm working on a new version of my smart contract I want to see how it's performing against another version of the same smart contract just a second to see where it is actually here you go anywho I can take the two versions I can do for example average gas used in the last seven days and compare how much gas each of those contracts used and I can see if the things I worked on on the new version are is actually optimized or even if it

uses more gas and I should deploy back to the old version the next one is protocol version comparison and this is actually really interesting for protocols the 0ex protocol for example I think in December moved from version to version 3 but because they here we go they cannot do like an instrument edition you could actually check how much of the protocol is used on each version sorry so here we can see for example that the version 2 of the protocol still has more usage than version 3 so not all users have actually migrated to the new version of the 0ex protocol the next one is incentive comparison for example so because we're

working in the incentive system you could actually check if your incentive mechanisms are good or in place here I actually checked if the 0x version 3 protocol gives more incentive to real layers to relay fill orders and here we can see that actually when people when real layers move to version 3 they stay there they stop providing version 2 version 2 support so the incentive is probably fine emitted events this is also one that's really easy to track so you could literally look at all of your contracts and I think this is also the 0x protocol because it's really nice to show and you can see which contract limits how many events per day for example and then we can see ok this

contract is used much more than the other ones maybe I should optimize it for this contract uses movies used much more here we see left but for example the exchange one is is the entry point to the 0 X system it's really used maybe we should optimize there maybe we should make it easier for other smart contracts to be used again permitted events you could also see in just a second you could basically also see top events so per event usage and then you can have outlier detection or you can for example see each event when it's fired we can see there's a lot of this wasn't withdrawals but we have a lot of it's much easier image yeah

a lot of transfers for example inside the protocol itself function calls as well and I find this one really fun to use so basically you see how each function even internal ones is called per transaction in I don't know a period of seven days again really good to find a place for optimizing really good to see what people are calling in which parts of your protocol or smart contract systems other people are using or your that outlier detection so here for example we can see each address how much is calling our smart contracts and then we can see for example someone trying to game our system so somebody's saying

thousands of transactions more than some other addresses so here we can see a couple of addresses sending three or four transactions in seven days but then we have one that's sending 1,400 transactions so it's probably trying either to do something or it's playing a really important part in your system but we can see that we don't have to we don't have to check it ourselves basically so and I don't know how to fool the screen this I don't know friendship so basically what I want you to take away from this is monitor from day one so when you open up your solidity file and start writing instantly start thinking about how this can be used how this can be monitored what are the metrics what do I care about thank you thanks I needed the translator not a Navy guy but this was

good as well so basically so monitor for day one the second one is monitor both business and technical metrics you all all of us here are on the business side of things not the technical side of things the technical side of things is a really really really bad thing that has to happen to actually generate some value the business is what it's actually important where things are important and then finally the most most important one is be proactive not reactive so thank you if anybody has any questions my name was then I work it tenderly thanks thank you very much it was very interesting does anyone in the audience would like to ask

any question mmm doodle raise your hand lovely thank you oh yes hey thanks for the talk maybe everybody's afraid of asking questions but I think I think I think like every everything you said is like true but it's also like maximalism kind of true right like of course we should have 100% coverage in our tests and of course we should have everything monitored from the get-go and of course we should have enough money to have like the big offices for all our engineers but this is a real world so like what how do you react like what's like the trade-offs you actually do in the actual field to get things done okay so as I

mentioned the reactive approach is like 60% of the team's and that what usually engineering things can things can do and this is the same type of talk as you have like with test-driven development where the guy comes there and say before even writing 50,000 lines of code for your application you should write a million a half lines of test code from your head and then just make those tests pass if this talk was open up ether scan every other morning or every morning usually people would go home and open it like once per week so I have to go the maximalist approach and then people if they take half of it this home it would be great in the real world the majority of things especially now unfortunately that and I'm really scared about this in the blockchain ecosystem is when things go wrong then people put stuff in place

for it to actually like not happen again that's the wrong approach what I would do for example will and what I actually do while developing is for the most mission-critical parts of the system everything is in place from day one so even before deploying everything is in place but for a less mission-critical stuff so for example I don't know checking your balance or whatever I will probably add the metric there if it's working or alert if it's working only once it fails so in the real world it's basically a balance of okay this can like create a lot of damage if it's not working or okay I really want to know if this is ringing the value to the business but if it's not that important and probably won't wanna train until it fails so that's the reactive approach I guess does anybody else like to ask any other

question no no don't be shy nice all right so thank you very much for coming give it our full molten yeah I think thank you and yeah I won't today [Applause]