Real-time monitoring of your validators. For you and the ETH network!

Transcript

[Music] hi everybody I'm Manny from Kane I hope you enjoy your party last night so Kim is a staking company we are running notes for Lido Ledger and other customers and currently we are talking about real-time monitoring of your SRM validator when you are not operator or even when you are a service taker it's really important that you are alerted when you miss the blog or when you miss attestation and that in real time and we are going to talk about it today

so basically we are going to talk about monitoring from the inside and entering from the outside then one two meter and if if you have sometimes uh we will I will take some questions um very quickly what is monitoring from the inside when you have a very nice validation infrastructure you have some execution now you have some constitutional Runners validator clients and so on and you can monitor very easily your CPU memory usage volume usage Network usage and this is really easily done by tools like promoters and grafana for example and here just a quick example of all our goator so I won't

focused on it today on monitoring from the inside but I will focus on monitoring from the outside so when you want to monitor you will node your validator client your validator case from the outside of course you need a very little infrastructure and basically this is an example of validator infrastructure you have some private subnets some public subnets so in private subnet you have for example an MAV boost with connecting some to some relays in public subnets you have your beacon your execution node and on private internet again you have your provided the clients rob3 Center replicated DB for anti-slashing and the world when you load your validator keys

here is a bad example don't do this of monitoring infrastructure so here we have a monitoring tool and we see that this monitoring tool is connected to the same Beacon to the same executioner than your validator clients you don't want to do this because you want your monitoring tool not to interfere in any way with your validation infrastructure for example here let's imagine your monitoring tool has a bug and does a lot of query on your beacon and executioner and in this case your editor client may have some troubles to connect to your beacon node and so you miss you may miss attestation and so don't do that what you can do for example is uh spin another

executioner and to plug it to your monitoring tool so your monitoring tool won't interfere with your bigger node and here you load the keys to monitor with just a file another um possible way is approximately the same but your meaning 22 is going to load public keys from webtociner so your web designer is your source of stress about what key you should monitor it's also possible we developed internally a tool when you're throwing tool which is called ethereum editor lecture and we open source it a few weeks ago and so I will talk about it right now so what monitor on the first thing you

want to monitor of course if when you missed a block when you miss a block it's not good for you it's not good for the network and so you want to be alerting in real time if you miss a Blog so for example here we miss a block at 12 we saw this here and we saw also a log on our monitoring tool and this is this corresponds to this Mist in Beacon chain and so possible issues when you miss a block is uh you may have an issue with the relay you may have not enough CPU on your bigger node or the get header may be sent too late or your validator key is not managed by any validator clients so you want to monitor when you miss a block in real time

second step is when new monitor means when attestation it's it could be normal sometimes to miss an attestation and so it corresponds to this in a beacon chain list and so the tool monitors Miss at the session it's a yellow curve here it's this smile layer with a start face it's indicates some some of our validators missed actually a lot misunderstation in this case and it's okay to miss one attestation it's less okay to miss two other session in a row and so it's the same here you have the orange curve and the tool indicates you when you miss two decisions in a row and it tells you in real time so you can variety and if you have a non-core guy

and if it is 12 o'clock in the morning the uncle guy can be can be alerted and also so you can miss one other session you may not miss two other session in a row but what is also possible is when you have too many of your attest session which are not optimally including normally when you attest for a block you at a session should be included in the next block so we call it inclusion distance and you should have a increasing distance of one and this this monitors actually the number of sub-optimal exercise Channel um you have for the last block and here for example this curve is interesting because it represents the loss of finality of ethereum network one month

ago the ring it was in May or on June May I guess and so during quite long period of time it is our prism and Equinox um yes had a lot of sub-optimal included that session and so you you should be alerted for that and it represents in the making chain this little number two here it's two that means we have two slot relates about intest intercession inclusion and so here we have a log Indica indicates that we had some non-optimal attestation inclusion next one is when you when you are not operator or even a service taker in

general you don't want to build the block yourself you want to build the block uh external Builder and external relayer and you want to know if your block has been built by a non-anoisted relayer or if your block has been built by your local execution client and so this curve indicates each time we propose the block not a built by a low listed relay in general it's because we have some real reorganization in our Beacon node and while we can also have some issue with nav boost for example and yeah also you have you want to know as an operator when you are going to

propose a block on the beacon chain you are you know two Apex in advance who is going to propose a block with a lot of keys on validator clients you don't want to reboot your register clients for example for an update or for anything you don't want to reboot it if you are going to purpose a block in The Next Step because you may miss a block so what is really important to know in advance I am I going to propose a block and if yes I should wait a little bit before rebooting my validator clients and basically what we what you see here is you are going to propose four blocks in the next two airbox and we know the web Keys actually

so we went in the ninth slot one in a certain slots and so on and so on and it's quite important for you also um okay so we talked about uh missed blog we talked about missed attestation and what we want to know also is is if you validator propose and sorry if you are a validator a test but for the wrong head for the wrong target of for the wrong source as you may know when you do a data session you have three main components which has which are Source Target and head and basically if you don't vote for the good for the good head you had zero zero way for that and if you vote for the

wrong target and wrong Source you can have even negative balance negative reward and really want to to know that because it's if you miss if you watch two too many times for the wrong uh head it could mean that you have an issue also with your foreign [Music] Target and header and also uh thanks to the reward API of the begin API here is it represents the theoretical um is there a reward for the last 20 24 hours

here in green is what you really had because you didn't act perfectly in Orange is the the Delta and yes it's a number represents this percentage so here we acted 99.5 percent perfectly it's not 100 so which would improve that and yeah or so another interesting um Matrix and when you exit validator or you want to be sure that your validator really exited because we assume when you trigger an exit you just trigger and you sign an exit message and you send it on your beacon normally there is no issue but it could maybe your bikini didn't correctly submit it the exit message on the network so maybe evaluator did not

exit and so yes with this feature you can monitor or you exited validator very briefly here is um yeah it's a total asset to manage here's the total assets of the return Beacon chain so epic the slots the number of your valuator slashed zero hopefully I hope the total number of network validator slashed uh uh your the number of your queued validators the number of total Network validators the validators the entry queue sorry entry queue length so it's quite long right

now so price of ease the Epic completion so it's from 1 to 32. the number of your active validators the number of total activities that are your share of validator and this is curve we already saw before and this is a logs the logs yes of The Watcher and it's also very responsive and you can get a very nice graph on your phone so thanks girlfriend and yeah so about monitoring electing so this tool can directly send message on Slack can direct me to send me some slack and send the metrics on parameters also write some

logs and so with promoters we can also set alerting based on Matrix so you can can be awakened up in the night and actually as a optional quite simple you just specify your bigger URL and execution URL a list of two key you want to watch or already stuff or a web designer URL if you want to use it a fair recipients if you want ah I forgot about this yes we want to be sure so that when you propose a block you propose it with a good Fair recipients yes very important I forget it so use the monitor also monitor if you propose a block with a wrong First Blood and it's compatible with lighters Nimble

sticker and prism not yet tested with a Roadster and yes basically that's it it's open source you can scan the QR code or you can go to the link and if you have some questions you have some times right now and after you can reach me on Twitter on by mail thank you very much for your time [Applause] if you have any questions you can raise your hand and I will bring you the mic foreign hello thanks for building this awesome

tool looks very great my question was for existing validators can you also sort of backfill his historical data with this tool well um actually no but it should not be very complicated to uh to implement because just right now it's it's a validator just you know wait for the next slot but we could add a feature just to you know to start for the start of the beacon chain and to backfill I basically it was really a real time on interim tour and after the usage move a little bit to more you know statistical stuff and uh and um yes API and more commercial metrics but yes I don't do this right now it's possible to do quite

easily and yes you don't think that the database I mean Primitives as a database but you don't need I don't know postgres Plug and Play yeah hello you recommended not to connect this to the existing Beacon chain node but it's very hard for solo stickers to spin up any other Get Rhythm just to connect this tool would it work it's also public knows or chain stack you can use chain stack if you want maybe you know change I guess is the only one RPC on the beacon chain before we had inferior that stopped the service a few months ago you can use handshake actually change stack use numbers and

there is some little feature that Nimbus does not support but yes you can use transcycle actually I'll reveal the olive oil company use chain stack resistor it works well hello what do you think of a different primitive with endpoint uh someone some better than other oscillators for example um can you repeat um what are some Primitives and points better than uh do you prefer a lighthouse Matrix or some other protocol I mean from it you you mean Primitives

or a beacon node endpoint for you know for sending some metrics but the tool is connected to a bigger node which is either writers or either prism or either Matrix no because we just use a I mean we just use a very um the open open Beacon API um I mean API and so lighters implements it and play some information implements it so just some minimal bugs with numbers right now but there is some tickets which are open and we use that all the time for the

question so it's over thank you very much have a nice day [Applause] [Music]