> I’m going to blame Open Telemetry. Yes, it’s powering the modern observability stack and yet I blame it for the mass confusion

The nice bit is this post will then lead to a bunch of companies’ log storage prices spiking and then a post about how you don’t need wide events trending.

A metric is a measure that can be aggregated over time. You can derive a metric from logs (count) but that doesn't make them the same thing. Spans have start and end times and parent pointer. Logs are arbitrary data, but can be correlated, can have application/org defined structure that's beyond the scope of the observability stack.

You could draw a class diagram with "Wide" - or whatever - as the base class, metric, span, log subclasses. Metric class requires an additional metric field, span requires start, end, parent, log is just open.

As someone who has been on that side of things for 10 years, can we stop building really complicated stuff that requires this pile of shit and dedicated staff to run it.

The whole otel, logging and metrics thing is based around "record everything" and dig through the pile of shit looking for things that might be helpful. Fine until you have 50TB of logs/traces online which is our problem. Oh that and the 50 gig prometheus instance. 99.99999% of all information we record is entirely and utterly useless and costs us money

Make simpler things, log only exceptional things, use the rule of silence.

wide events feel like structured logs, great for high cardinality data but not so much for time series

(2024)

prev. submission - https://news.ycombinator.com/item?id=39529775

Very ignorant take. If all of us had infinite infrastructure there would indeed be no reason to separate the three at generation, but for those that have to actually count the costs, paying for servers to store that amount of duplication is real issue, and even more (for some reason, by order of magnitude or more) if you're using one of big observability providers.

It is divided mostly because database to store metrics, and database to store essentially freeform documents are very different products and need different features from query language, and having "just metrics" allows to store them with compression reaching 1byte per metric data point or less.

Logs and traces I admit, split here is unnecessary, and it should be just presentation difference. But metrics is often just unrelated to any single event summary of the state of the system. Sure you can just log every single log entry with request metadata and get request rate, percentiles etc. by calculating the events, but that is just too wasteful for most.

Meh. This is just structured logging with some post processing. You can definitely derive metrics and traces based on them, but doing so is more expensive in many ways.

Potential PII nightmare. You're now potentially circumventing access controls from sensitive data.

Why is 77rushi77's comment flagged + dead? Seemed like a good question.

Edit: Looks like it's been restored.

The Scuba example is convincing for debugging. What the post doesn’t cover is cost. If you keep raw wide events instead of pre-aggregating, how do you handle storage and query cost at scale? Is sampling the main lever, or do teams still keep cheap metrics for alerting and use events only for investigation?