Remix.run Logo
Veserv 7 hours ago

Saying they are all the same when no fidelity is lost is missing the point. The only distinction between logs, traces, and metrics is literally what to do when fidelity is lost.

If you have insufficient ingestion rate:

Logs are for events that can be independently sampled and be coherent. You can drop arbitrary logs to stay within ingestion rate.

Traces are for correlated sequences of events where the entire sequence needs to be retained to be useful/coherent. You can drop arbitrary whole sequences to stay within ingestion rate.

Metrics are pre-aggregated collections of events. You pre-limited your emission rate to fit your ingestion rate at the cost of upfront loss of fidelity.

If you have adequate ingestion rate, then you just emit your events bare and post-process/visualize your events however you want.

m3047 4 hours ago | parent | next [-]

Good summary IMO.

> You can drop arbitrary logs to stay within ingestion rate.

Another way I've heard this framed in a production environments ingesting a firehose is: you can drop individual logging events because there will always be more.

otterley 3 hours ago | parent [-]

It depends. Some cases like auditing require full fidelity. Others don’t. Plus, if you’re offering a logging service to a customer, the customer’s expectation is that once successfully ingested, your service doesn’t drop logs. If you’re violating that expectation, this needs to be clearly communicated to and assented by the customer.

The right way to think about logs, IMO, is less like diagnostic information and more like business records. If you change the framing of the problem, you might solve it in different way.

otterley 7 hours ago | parent | prev [-]

> If you have insufficient ingestion rate

I would rather fix this problem than every other problem. If I'm seeing backpressure, I'd prefer to buffer locally on disk until the ingestion system can get caught up. If I need to prioritize signal delivery once the backpressure has resolved itself, I can do that locally as well by separating streams (i.e. priority queueing). It doesn't change the fundamental nature of the system, though.