I think there’s too many expectations around what logging is for and getting everyone on the same page is difficult.

Meanwhile stats have fewer expectations, and moving signal out of the logs into stats is a much much smaller battle to win. It can’t tell you everything, but what it can tell you is easier to make unambiguous.

Over time I got people to stop pulling up Splunk as an automatic reflex and start pulling up Grafana instead for triage.

We arrived to that too, Grafana does a lot of heavy lifting in localizing most of the issues.