Monitoring Validators

Transcript

hello everyone today i'm going to be talking about how to monitor your validators with lighthouse so the solution i'm going to be talking about has no requirement for third-party hosted services your nodes your data it is capable of remote monitoring so you can monitor from anywhere in the world you don't have to be inside your staking environment you can monitor non-lighthouse validators or validators that aren't your own you can get very detailed and evolving information uh from this method as well so this method is intended for a technical audience of ethereum stages it's a video about me my name is paul hana i'm a director of sigma prime i'm also a lead developer on lighthouse which is a uh ethereum consensus aka f2 uh implementation written in rust

maintained by sigma prime so they can contributed to that protocol specification since its inception and i'm also responsible for about 60 000 validators spanning 35 nodes across three networks so the vast majority of those validators practically all of them are testing validators we don't have that much effort so the validator monitor is built into the lighthouse beacon node so you enable it with one of the three flags on the command line so these three flags they all have kind of two effects one of them is enabling the feature and the other is specifying which validators it should be monitoring so the first flag validated monitor pub keys just takes a comma separated list of public keys by the cli the second one validator monitor file

is basically the same thing except it reads that comma separated list from a file it's good if you have lots of validators and the third is a validator monitor auto so what that does is uh validators that use what when it notices particular validators using a particular endpoint um it'll then enroll enroll them for the validator monitor so the idea is that um if you have validators directly connected to that beacon node it will automatically monitor those validators so that one is not so useful for remote monitoring that's really useful for insider staking environment so before you start using this feature just make sure you read the docs because there's a couple more flags you want to add as well just to make sure you're

getting all the data so what it does is for that set of validators it produces info messages when the validator is performing well uh it produces warn or error messages when that validator is not performing well and it also exposes a much more detailed per-validated metrics to prometheus and therefore grafana so like i said you can monitor any validator even those that you don't own or those are running inside a different implementations validator client and you can watch them in real time by the logs or metrics so beacon chain does already do a fair bit of this but it's a pretty diffic different solution really um so big change is great i got a lot of respect for those teams they've been

really helpful to us developers but theirs is a very user friendly block explorer um with a slick web ui that would be i imagine quite difficult to maintain whereas this is like you know just a bare bones cli uh tool that we can modify really quickly we as core developers use it for troubleshooting issues and for you know quickly quickly adding quite detailed things without having to worry about um you know the average user very much um and it also has a nice benefit against beacon chain is that it's not third party so it's your node and and your data which is pretty appealing to some people so the validated monitor and vegan chain

excuse me are both tools they're both great and they're not exclusive and you should use them together so the validator monitor is also a communication tool so it allows us developers to communicate to you stakers about the performance of your validators without us knowing which validators you are or revealing your identity um so we have a problem um sometimes where we can see validators on mainnet not performing well but we don't know we can't we can't really contact the stages we can't you know resolve and validate a public key to a phone number or an email address so really have no way to inform people um so this is a way that as we learn with problems on mainnet we

add them to the validator monitor add new warnings for them and then that allows us to to give you a bit more of diagnosis about how your validators are performing without us actually knowing what your validators are so it's really useful in that perspective so the people that should use it i think people with a very high technical proficiency it is useful for average users but i think um to appreciate all the features you have to be pretty technical so i would suggest staking pools ethereum ecosystem developers perhaps highly technical users and also non-lighthouse users i mentioned this before but um if you're say staking pool using another client it might be quite useful to run lighthouse or another client um tomorrow's your validators as well just in case uh it's

picking up on something that the other implementation is is not getting so i think people shouldn't be afraid to trade out afraid to try it out um if you're just running on your laptop you know away from all your safety infrastructure there's nothing bad that can happen the worst thing really is that you might just get a bit confused so the user interface for this tool is pretty simple uh configured by cli i talked about those three flags before uh and then what it does is just print logs messages and then expose metrics by prometheus so for the rest of the talk i'll go into um i'll show you some of the log messages and prometheus metrics so the information we include in these logs you get real-time data so that's when you get blocks attestations exits

and slashings and you get a little log for those and we also do end of epoch summary data so when our consensus engine is computing rewards and penalties we will spit out little logs to say like exactly not exactly but more detail about what your validator did during that epoch we give a bit more valid a bit more information than beacon chain does which is quite cool we also create error logs um when your validator is doing the wrong thing which can be very helpful especially when you are going back trying to figure out perhaps beacon chain has told you that your validator didn't do well and you want to try and figure out exactly why keeping these logs around can be really useful just to go back and track through them and figure out why so here's a bit of an example

of some logs of validator zero just watching that validator um and we can i'll just kind of walk through the i guess a bit of a flow of an attestation for it so this is like one stream of logs and i just cut the middle out of it because it wasn't very interesting so we can kind of see at the top we see the attestation was published and then we can see the attestation being included in several aggregates and then we can see the attestation being included in a block and then later time goes on probably e-book two later we'll then see at the bottom rewarded by protocol so then we get a notification here saying that we've processed the rewards and penalties for that validator and we can see those attestations before now enshrined in the canonical chain and

they'll be rewarded for it we can see there the validators match the head match to target matches source they've done really well and up on the other side here's a set of logs where things didn't go perfectly so we've got the a log here the reward by protocol kind of log it's the same same classification as the last one in the previous slide but we can see here we didn't vote for the correct head so matched head is false and after that we get a little warning log to say the attestation fails for matched head so this can be like i said really useful when you're trying to figure out what went wrong with an attestation so we'll quickly have a look at the metrics as well um i don't want to go too far into them but this is kind of one part of this metrics dashboard i've

got a link to it later in the slides so here we can see how many values we're monitoring their balances and then we can see the canonical inclusion distance of which they got their reward for and then we can also see these um some of the metrics are followed by goods so there's three of them in a row you know attestation success um head hit and head and target hit so if you're getting messages if you're getting events in the good column that's good and then underneath we can see the inverse of each so gestation misses head misses and target misses that's showing us when we're not quite um voting for the right blocks and not getting ideal rewards then a little bit more details

we can see in these events we've got these real-time events counting when we see a station on the network from our validator and aggregate blocks when we see the validators attestation included in aggregates and then when we see those attestations get it included in blocks um once again useful for debugging and then something i find very interesting the second part is uh a bit about timing delays so we're looking at the delays between whether the first moment in which a validator should be publishing it's um a message and when we received it so this can be really useful for debugging things like it's a dodgy network in your staking infrastructure or perhaps an overloaded node that's um not able to keep up with

the network or published messages quickly so a couple of quick caveats about this system it requires a synced node pretty straightforward um it does require it does it will occasionally report on non-canonical change because it's reporting as it goes on every block if there's a fork or a weird orphan block it might create a log that doesn't make it into the canonical chain not a big deal just something you've got to be aware of when you're double shooting it's intended as a diagnostic tool so it's not supposed to be you know like a rock solid perfect monitoring tool it's for diagnostics um and just be aware it is like i said before intended for a technical audience so something to be aware of um got a few links here so a couple of guides docker setup

the dashboard i mentioned earlier and our discord if you have any questions so i think that's my 10 minutes up so thank you very much and i hope you enjoy the tour thank you