| ▲ | lanstin 39 minutes ago | |
cheaply dropping log msgs you cannot handle is absolutely essential for an observability system - otherwise excess load can take down the logging infra which can (if msgs aren't dropped) take down the prod network/app trying to send reliable log msgs. Audit logs are a distinct feature. | ||
| ▲ | otterley 15 minutes ago | parent [-] | |
That's fine if you plan for it and can clearly delineate which logs can be dropped and which can't. The challenge is that most applications log to a single stream that consists of both high and low-priority logs muxed together and is sent to a single destination, making it impossible to distinguish the two, and there's only one receive buffer. Excess load can't take down a well-engineered log collection infrastructure. There can be overload, but the backpressure should propagate downstream and senders and intermediaries should buffer locally if needed. Once the collectors are able to catch up again, the spooled messages will be dispatched, and the backlog should recover. A well-engineered logging system for sites that care about integrity and durability should look a lot like a distributed message queue. > Audit logs are a distinct feature. In my experience, this is not always as distinct as one might hope. On multiple occasions in my career, a customer demanded we perform research using our logs to answer, and the information they sought were not in the class of logs that were considered "audit logs" in advance. Everyone chooses differently what qualifies as "audit logs"; it doesn't have an objective definition. | ||