Sébastien Rannou - Real-time monitoring of your validators. For you and the ETH network!

Transcript

🔍 Real-time monitoring of Ethereum validators, including inside and outside blockchain monitoring, provides access to various metrics and signals but can be noisy and tricky to make sense of, requiring client diversity at scale.

good afternoon uh the next speaker is Sebastian ranu and he'll tell us everything about real-time monitoring of our eat validators thanks hi um so yeah my name is Sebastian I work at k um which is a provider of eum validators so we work with li with Ledger as well and and other and yeah we'll be discussing a bit about how to monitor eum validator at scale in real time um so yeah first we'll discuss a bit about monitoring from the inside then about monitoring from the outside using the blockchain uh we then have a quick look about the different signals

you want to Monitor and maybe we'll have some time for questions at the end um so monitoring from the inside so the idea about monitoring from the inside is like you are running an infrastructure of ethereum and you have access to all the components from the inside so you can directly measure uh different metrics such as like resources or also like well CPU memory volume networks um it kind of like depends on your setup if you're using kubernetes you will have maybe different sets of uh of signals here um so yeah I've listed here the typical components of an eum stack so you have like execution nodes which are used to produce blocks um you also have like consensus nodes such as

well Bon nodes basically rightor clients we signer M boost and so on um and so inside is that you are going to get for free a lot of of signals so it's going to be a bit noisy and making sense out of it is a bit uh is a bit tricky um but as you are getting a lot of information you like it's very helpful to to help debugging and deep diving into like very specialized cases whenever they happen another disadvantage of monitoring from the inside is that uh if you are like working at scale you probably want to incre encourage client diversity so you're probably going to run different types of execution nodes consensus nodes and so on and those nodes are going to expose different metrics so for instance

🔍 Ethereum VAR Watcher is a monitoring tool that connects to execution and beacon nodes to monitor validators in real-time, providing accurate information about the performance of public keys.

the definition of the last block scene by a bon chain like te may be different than the one used by Prism and so if you try to aggregate those data together they may not make sense um so this is where monitoring from the outside comes and the idea about monitoring from the outside is that instead of looking inter signals you can directly look on chain what's going on so you can look at the content of blocks and Slots try to extract information from there and in Fair how your public keys are performing um so like this is just a small uh diagram of right the previous uh previously mention um components of eum stacks but so yeah you have on the left MV boost which is typically used like for M and it's connected to

different Rel layers um you have a pair of beon and execution node uh to which a client con next and so the client which loads validation Keys is going to subscribe to the Bon which in turns will subscribe to the m boost to get new proposals and so um if you have like um yeah internal monitoring you will have like basically your monitoring um um connected to all of those and watching different signals and so outside monitoring on the other hand uh is like you introduce a monitoring tool and you connect it directly to the extion node and the Bon node and from there you are able to look at what's happening on the blockchain so you actually don't want to connect it directly to the infrastructure you are using for validation Keys uh because uh well um let's say I don't know the Bon node is

told or something uh you won't be able to get information about new slots and so you won't be able to monitor so you want actually to have two separate Stacks here you want to have one stack for uh the validation path and one stack for the monitoring itself um so this is uh yeah what I just mentioned so yeah the idea is just to like run a separate stack for entering uh and then uh so at K we developed a tool called the etherum VAR Watcher it's a small python um tool which can connect to execution nodes and beon nodes and you can configure it with a list of public keys and then it's going to look for different seges we are going to discuss a bit about later um and yeah it's open source so like if you want to

to give it a try you like happy to yeah would be happy to to have users um and so um the tool can be configured either by giving a list of public keys or it can also be configured by directly connected to weigner instance and on weigner there is a a small um RPC call to get the number well the list of keys that are handled by by web deser um so that's yeah that's basically it um so yeah let's discuss a bit about the different signals you want to watch for um so the most obvious one is a m blocks um so like proposing a block is is at in the lifetime of validator key something

👀 Real-time monitoring of validators is crucial to catch missed blocks and attestations, and outside monitoring can help detect potential issues beyond your own infrastructure.

quite rare like it happens once every four months on average I think these days uh and uh whenever you propose it it's going to like correspond to all of the execution reward you you have for the four months so you don't want to miss it um and uh yeah there can be multiple reasons for missing a block uh there can be an issue that does not depend on your own infrastructure uh for instance if you have like some issues with external Rel layers um and this is also where outside monitoring is more like is quite interesting because you will also catch um issues that are outside of your U uh infrastructure um so yeah we expose like M blocks um you also want to have a look at like M attestations um so like this

happens every well your vator has to propose a block sorry has to sign an attestation every six minutes on average um and it's it's quite like if you operate at scale you will often have a small tail of like mestation uh due to network hicups or this this sort of things so this is quite a noisy signal but it's quite important to uh to have it at scale so that like let's say like if 70% of your infrastructure is missing attestation there is probably going something going on um so this signal is noisy but uh can be useful um yeah when running at scale um for issues that are related to very specific key double miss attestations that is like missing two

attestation in a row is is a bit more interesting uh it can like for instance happen if uh if you misconfigure your your validator or if it's not loaded properly by the client you will you will have a uh yeah you will be able to to alert on this so basically yeah you have like two sorts of alerts on attestations at scale by looking at single M attestations and pair validator Key by by looking at double m attestations and missing attestations yeah I've listed a few issues possible Ro causes here but there are others um another uh signal is uh so on whenever a validator signs for an

attestation uh it's like included in a slot and there is something called the inclusion distance which is like the distance between your attestations and when it's included on the on the Bon chain and the higher the the inclusion distance is the lower the reward you are going to get so it's also a signal you want to to look for uh so we also expose it here and yeah on like same as before like this can happen because I don't know you you probably like maybe you have like network issues with your bon and it's not like propagating enough attestations or it can also be like a network event where uh um s aggregators or other actors of the network are not like getting attestation in time and so

👀 Real-time monitoring of validators involves tracking inclusion distance and Source Target head to optimize rewards and effectiveness on the ETH network, as well as watching for signals indicating if a relayer is not replying in time and checking if the proposed block was built locally or using a relayer.

like this graph here is like what happened when we had some finality issues a few months ago on the inter Network um another signal you want to watch for is a untimely or bad Source Target head um so it's a bit similar to the previous one um you you you you won't get the maximum U theoretical uh number of like consen layer reward if you don't vote for the right Source Target or head and so it's also a signal you want to to Monitor and we are using like the rewards API API sorry which was introduced um uh I think a few months like six months ago or something like this on the Bon RPC API and you can um

quite easily uh know how much you could have the theoretically made if you were like doing the perfect job versus like how you are actually performing and you can get an idea of your Effectiveness which is also the signal used by Bon chain beon Shad um um so that's uh oh yeah another another important signal to watch for is like if you using me uh so the way it works is like uh the beon is going to ask like whenever one of your rator key has to propose the block the Bon is going to ask to the relayer typically via M boost to get a header and if the reer does not reply in time uh then the

beon is going to fall back to build a BL locally using your local execution node uh and the public M Pool and so this signal here is like uh connect to relays and and checking if the block you just proposed is actually a block you build locally or using a relayer so it can um give you an IDE if like you misconfigured me or or something like this um yeah uh so another quite interesting feature is like so it's not like um a signal like that shows performance issues or whatever but it's on the Bon chain you can know two EPO in in advance if one of your validator is going to propose a block and it's quite an

👀 Real-time monitoring of validators helps anticipate and address block proposal issues, and comparing network metrics with individual validator metrics can be done through a python script and dashboard.

interesting metric because when you operate at scale you while you have upgrades to perform you have to reboot validators and so on so by looking in advance if there's something that is going to be proposed in the next two blocks uh next two EPO sorry you can decide or not to to take this action and it's quite like a a simple way to uh to increase your u u well to decrease your your chances to to miss a block um yeah um so if you are a big node Runner you probably also have like Automation in place to exit validators um so it's yeah just a simple metric to to look for as well um and here's another one so this screenshot is is taken on holes key

where we are one of the operators um so all of the signals I mentioned before can be a bit noisy on some networks due to the scale or like external reasons external factors that may not be in your responsibility and so if you are looking if you are seeing like high misses of attestations and all it's it's always interesting to compare it with how the network is doing to know if it's actually something that happens because you are badly operating or because it's like something is going on on the network uh and so we also have the ability to watch for all keys of the network in real time export those matrics and be able to compare them with the set of matrics uh that are just for our validators um and uh yeah so as I said like we have

like this python script that exports metric to pretus and then you can like we also have a dashboard a graph dashboard which shows all of those metrics in a in a nice way so this this is bit old because it was like a few months ago where the NQ were much much much higher than they are today um but if they ever like if the number of activators activators increases suddenly we will be like you can also like track the ENT q and exit Q times here and there is also some other information like the AUM you are managing or like whether or not you are slashed which hopefully stays to zero

um so yeah as I said it's like using promerus so if you have alerting on top of it you can for free get promerus alert um and you can also directly connect it to a slack Channel and it will output to you like whenever like something is going on on your uh public Keys uh and you can also like have some information from the loog files um and yeah some some things we want to add soon um so we want to be able to add different labels per key so the idea here is to say like oh this keys I don't know running on a specific region or data center with like prism or lighthouse in this very specific version and then the tool can export it as um

🔍 Real-time monitoring of validators with alerting, Slack integration, and upcoming features like adding labels for comparison of different versions and locations.

promit use labels and then you can compare I don't know how this specific version of prism is doing compared to this new one and it can help a lot with qualifying new releases or like Network cups of I don't know running in Brussel versus Ireland or this kind of stuff so it's not a feature we have yet implemented but we are working on it and we think we'll have it soon and another one is the ability to backfield alter data that is like to run the tool against past run on the beon Chain uh and there are tricky ways to do it with Pro materal but we think we we can do it and this would be helpful for instance for postmortem analysis where you want to integrate a new signal and you want to see how it was performing at the time

👀 Real-time monitoring of validators is in progress, with plans to improve performance and backfill data for analysis.

um yeah I think that's uh that's what I had to say if you have questions um so yeah there is a link to the open source version of the ACM VOR Watcher which was initially written by Manu who is not there but yeah feel free to to scan the QR code to get uh to get access to it and yeah if you have questions happy to take them questions hi um thanks for the presentation really nice um you said you use the Bon API rewards um endpoint that was introduced

some months ago how do you plan to do that with all the data because I think it's if it's slow with I don't know if you have any plan already or something U sorry so the question is like how how fast it is like to query this data or yeah because right now that Endo I think it's really fast with um data from finalized to the Head yeah but I think with all the data it's not as fast I don't know if you already have for now yeah we are not like so right now like the tool is only running real time so it doesn't have to go in past ranges so um yeah I guess this is something we have to figure out if we want to pass data I was just trying to know if you had any plan for the Back Field fure um not yet like we have

aggregators like not not here but like internally which are like being written to um so yeah I guess one one possible way just to is just to look at the effective balance no no sorry oh sorry as the active balance you can like kind of like get an idea of the performances by taking snapshots of like your active balance at the end of the day from beginning of the day do a sub substruction and some math uh but it's not going to to give you fine grain signals so yeah I guess we'll have to to find a way to make it fast if it's too slow like do you have an idea of the order of magnitude like is it like 10x slower or well I have a bit of experience using this end points and it's a bit experimental as you said yet

👨 Real-time monitoring of validators is experimental, with some issues using Lighthouse or Prism, but the speaker will discuss it further in the next talk.

so I also have some issues with a back feeling are you using Lighthouse or prison corre lious maybe we can discuss it now in like yeah let's let's sh thanks a lot for for the answer other questions no okay let let's thank the speaker again and see each other for in about 10 minutes for the next talk