The big idea

What is a logging pipeline, specifically?

Not the dashboard, not the search box — the unglamorous plumbing before either of those exists: how do log lines from thousands of running service instances get collected, buffered, structured, and delivered somewhere durable, without ever becoming the reason a service itself slows down or falls over.

We'll build the write path: local buffering that decouples the app from the pipeline's health, backpressure with an explicit policy, batching, structured parsing, delivery guarantees, cost-controlling sampling, and — the sharp edge every real system eventually hits — protecting against a logging storm caused by the very incident you're trying to observe.

How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the diagram grow, hover the boxes, and at the end kill the queue and trigger a logging storm to see what survives. Hit Begin.

Step 1 · The baseline

Write straight to the log server

Simplest version: every log line is written synchronously, directly, over the network to a central log server the instant it's produced. What's the danger?

Design decision: A synchronous, direct write to a central log server for every log line. What's the danger?

The call: If the log server is slow, unreachable, or overloaded, every application making a synchronous write is now blocked on it too — the logging system, which should be observing the application, can directly cause an outage IN the application. — Coupling the application's hot path to a synchronous network call for every log line means the log server's health becomes the application's health. An observability tool that can take down the thing it's observing has its priorities backwards.

Synchronous, direct writes couple the application's own request path to the log server's health — if the log server is slow or down, every application doing a synchronous write is now slow or down too. A logging pipeline's first, non-negotiable rule: never let logging become a dependency the application blocks on.

Observability shouldn't be a liability: The entire point of a logging system is to help you understand and debug production issues — it should never itself be a source of new ones. Every subsequent design decision in this pipeline traces back to protecting that one invariant.

Step 2 · Decouple with a local agent

The app writes locally, an agent forwards

Fix Step 1: the application shouldn't talk to the network log path directly at all.

Introduce a local Log Agent — a sidecar or daemon running alongside the application. The app writes to a local, fast destination (stdout, a local file, a local socket) and returns immediately; the agent independently tails that output and forwards it asynchronously. The application's request path is now completely decoupled from the health of anything downstream.

Decouple with a local hop: A local write (disk, memory, a local socket) is orders of magnitude faster and more reliable than a network call, and failure-independent from anything remote. Inserting a local agent as a buffer between "produce the log" and "deliver the log" is the same decoupling pattern used everywhere producers shouldn't wait on consumers.

Step 3 · Backpressure and bounded buffers

What happens when downstream can't keep up?

The agent forwards to a durable queue. That queue — or whatever's past it — gets slow or falls behind. The agent's own local buffer starts filling. What should happen when it fills completely?

Design decision: The agent's local buffer is full because downstream is slow. What's the right policy?

The call: Use a BOUNDED buffer with an explicit overflow policy — spill to local disk first, and if even that fills, drop the oldest (or least important) entries rather than blocking the application or growing without limit. — A bounded buffer with a clear, deliberate policy (spill to disk, then drop-oldest under sustained pressure) accepts that during a severe or prolonged downstream outage, SOME log volume may be lost — but the application itself is never blocked, and the host never runs out of resources. This is an explicit, acceptable trade-off, not an accident.

Give the agent a bounded, disk-backed buffer with an explicit overflow policy: spill from memory to local disk under pressure, and if the buffer is still full after that, drop the oldest (or lowest-priority) entries rather than blocking the application or growing without limit. Some log loss during a severe, sustained outage is an acceptable, deliberate trade — an application outage caused by its own logging pipeline is not.

Backpressure needs a decision, not a default: Every buffered system eventually has to answer "what happens when the buffer is full" — block the producer, grow unboundedly, or drop. For a logging pipeline specifically, given the Step 1 principle, drop (with disk-spill as a first line of defense) is almost always the right default, unlike systems where losing data is unacceptable.

Step 4 · Batch for throughput

Don't send one network call per line

Forwarding every single log line as its own network write to the durable queue is enormously wasteful of connection overhead at real log volume. What's the fix?

Batch log lines — accumulate either N lines or T milliseconds of output, whichever comes first, then send one batched write. This amortizes network and write overhead across many lines, at the cost of a small, bounded delay (milliseconds to a couple seconds) before a given log line is actually durable downstream — an easy trade for the throughput gain at real scale.

Batching, again: The same batch-vs-per-item trade-off that shows up in ride matching (batch requests) and hotel pricing (batch recomputation) applies here to the write path itself: a small, bounded latency cost buys a large efficiency gain, because the marginal cost of "one more item in an existing batch" is far lower than "one more independent operation."

Step 5 · Where does parsing happen?

Raw text vs. structured fields

A raw log line is just a string. Downstream search, alerting, and dashboards need structured fields — timestamp, level, service name, trace id. Should parsing happen at the agent (distributed, at the edge) or centrally, after the queue?

Design decision: Turning raw text into structured fields — parse at the edge (agent) or centrally, after the queue?

The call: Centralize structured parsing in a dedicated service reading from the queue — one place to define, version, and fix parsing rules, even though it becomes a component that must scale with total log volume. — Centralizing parsing means one codebase, one place to add a new log format or fix a parsing bug, and one thing to monitor for parsing errors — the trade-off is that this component now has to scale with the FULL aggregate log volume, which is a real, but very solvable (horizontally scalable) capacity planning problem.

Centralize structured parsing in a dedicated Structured Parser service reading from the durable queue — one place to define and evolve parsing rules across every log format, at the cost of a component that must scale horizontally with total log volume (a well-understood, solvable scaling problem, unlike inconsistency across thousands of independent edge parsers).

Centralize what needs consistency; distribute what needs scale: Parsing rules benefit from being defined and fixed in ONE place; the raw collection/buffering work (Steps 2-4) benefits from being distributed to the edge, close to each application. Recognizing which property a given piece of the pipeline actually needs is what determines where it should live.

Step 6 · Delivery guarantees

At-least-once, plus dedup

A batch fails to deliver — a network blip mid-send. The agent retries. What if the batch had actually landed the first time, and the retry creates a duplicate?

Design decision: A delivery might succeed even though the sender thinks it failed and retries. How do you avoid double-ingesting?

The call: Retry aggressively (at-least-once delivery), and attach a unique, idempotent key to each batch so the downstream store can detect and discard true duplicates on arrival. — At-least-once delivery (retry until acknowledged) guarantees nothing is silently dropped due to a transient failure, and pairing it with an idempotency key (a deterministic id per batch) lets the downstream store recognize "I've already ingested this exact batch" and discard the duplicate rather than double-counting it.

Use at-least-once delivery — retry on any uncertain outcome — paired with a deterministic idempotency key per batch, so the downstream path (via the Dead-Letter/Retry store for anything that needed a retry) can recognize and discard a true duplicate rather than double-ingesting it. This achieves effectively-once ingestion without needing distributed transactions.

At-least-once + idempotency ≈ exactly-once, without the coordination cost: "At-least-once delivery, deduplicated by an idempotency key at the consumer" is the standard, pragmatic pattern for effectively-once processing across unreliable networks — it's simpler and more robust than trying to guarantee exactly-once delivery at the transport layer directly.

Step 7 · Sampling at high volume

You don't need 100% of everything

A fleet of thousands of instances emitting verbose debug-level logs can produce a genuinely enormous, expensive volume — most of it routine and never looked at. Should the pipeline ingest all of it?

Design decision: Ingesting 100% of debug/info-level logs at fleet scale is expensive and most is never read. What's the fix?

The call: Keep 100% of high-signal logs (errors, warnings) but sample only a representative fraction of routine, low-signal logs (debug/info) — preserving the ability to debug real problems while controlling the cost of the flood that's rarely useful. — Errors and warnings are exactly the events you can't afford to miss, so they're never sampled away. Routine, high-volume, low-signal logs are the right target for sampling — even keeping 1-5% of them still gives you a statistically useful picture of normal behavior at a fraction of the ingestion cost.

Introduce a Sampler: always keep 100% of high-signal logs (errors, warnings, anything tagged critical), but sample a smaller, representative fraction of high-volume, low-signal logs (routine debug/info). This preserves the ability to debug real incidents fully while controlling the cost of ingesting the much larger volume of routine noise.

Not all signal is equally valuable: Sampling isn't "losing data" indiscriminately — it's recognizing that error-level events are rare and precious (never sample them away) while routine events are abundant and individually low-value (a representative sample captures the useful statistical signal at a fraction of the cost).

Step 8 · The sharp edge

The logging storm

An incident hits: a retry loop somewhere starts logging an error on every failed attempt, at high frequency, across many instances simultaneously. Log volume spikes 100x in seconds — right when you most need the pipeline to be reliable and fast.

Add per-service rate limiting on the logging path itself, distinct from the sampling policy: if a single service's log volume spikes far beyond its normal baseline, cap it (drop excess volume beyond the cap, optionally keeping a representative slice) rather than letting it saturate the agent's buffer, the queue, and potentially the app itself via the Step 3 backpressure policy. This is the final application of the Step 1 principle: the observability system must never amplify the very incident it exists to help you debug.

Design for the unhappy path: Queue outage → agent buffers locally, app unaffected. Duplicate delivery → idempotency key dedups it. Logging storm → per-service rate limiting caps the blast radius before it reaches backpressure's drop-or-block decision. A pipeline that only works at normal log volume is a demo; one that survives its own worst-case volume spike is a product.

You did it

You just designed a logging pipeline.

• S — y
• A —
• A —
• B — a
• S — t
• A — t
• S — a
• P — e