Remix.run Logo
otterley an hour ago

On the other hand, traditional syslog is UDP based, so as soon as the receiver experiences CPU or I/O starvation and its receive buffer overflows, it will begin dropping messages. That's not great for observability, and may well be impermissible at many sites that need end-to-end log integrity (e.g. audit logs).

skullone 26 minutes ago | parent | next [-]

You can use TCP and other mechanisms for guaranteed delivery. Like, almost all LIDR logging across the planet uses TCP syslog, which is durable and attestable to in courts.

lanstin 39 minutes ago | parent | prev [-]

cheaply dropping log msgs you cannot handle is absolutely essential for an observability system - otherwise excess load can take down the logging infra which can (if msgs aren't dropped) take down the prod network/app trying to send reliable log msgs.

Audit logs are a distinct feature.

otterley 16 minutes ago | parent [-]

That's fine if you plan for it and can clearly delineate which logs can be dropped and which can't. The challenge is that most applications log to a single stream that consists of both high and low-priority logs muxed together and is sent to a single destination, making it impossible to distinguish the two, and there's only one receive buffer.

Excess load can't take down a well-engineered log collection infrastructure. There can be overload, but the backpressure should propagate downstream and senders and intermediaries should buffer locally if needed. Once the collectors are able to catch up again, the spooled messages will be dispatched, and the backlog should recover.

A well-engineered logging system for sites that care about integrity and durability should look a lot like a distributed message queue.

> Audit logs are a distinct feature.

In my experience, this is not always as distinct as one might hope. On multiple occasions in my career, a customer demanded we perform research using our logs to answer, and the information they sought were not in the class of logs that were considered "audit logs" in advance. Everyone chooses differently what qualifies as "audit logs"; it doesn't have an objective definition.