The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is a logging pipeline, specifically?
Not the dashboard, not the search box — the unglamorous plumbing before either of those exists: how do log lines from thousands of running service instances get collected, buffered, structured, and delivered somewhere durable, without ever becoming the reason a service itself slows down or falls over.
We'll build the write path: local buffering that decouples the app from the pipeline's health, backpressure with an explicit policy, batching, structured parsing, delivery guarantees, cost-controlling sampling, and — the sharp edge every real system eventually hits — protecting against a logging storm caused by the very incident you're trying to observe.
What the new pieces do
- Applicationclient
- A service instance producing log output — potentially thousands of these running concurrently across the fleet.
Step 1 · The baseline
Write straight to the log server
Simplest version: every log line is written synchronously, directly, over the network to a central log server the instant it's produced. What's the danger?
A synchronous, direct write to a central log server for every log line. What's the danger?
A single log line is lightweight, but a SYNCHRONOUS network write per line means the application's own request path now depends on the log server's latency and availability — that's not lightweight at all when the log server is slow or down.
Coupling the application's hot path to a synchronous network call for every log line means the log server's health becomes the application's health. An observability tool that can take down the thing it's observing has its priorities backwards.
Disk usage is a real, separate concern (addressed later via sampling), but it's not the IMMEDIATE danger of synchronous writes — the immediate danger is coupling application latency/availability to the logging path.
Synchronous, direct writes couple the application's own request path to the log server's health — if the log server is slow or down, every application doing a synchronous write is now slow or down too. A logging pipeline's first, non-negotiable rule: never let logging become a dependency the application blocks on.
Why this piece earns its place
The failure usually arrives as slowness rather than as an outage, which is what makes it so easy to wave through in review. A log server under load answers in seconds instead of refusing, so every request handler that emits a line sits there holding its thread, its connection and whatever locks it took on the way in. Concurrency drains, the accept queue backs up, and the service is effectively down — with healthy dependencies, passing health checks, and nothing in its own logs to say why, because the logs are exactly what is stuck. The defense is not a shorter timeout; it is removing the wait. And the cheap-looking half-measure — keeping the network write but pushing it onto a background thread inside the process — does not remove it: that in-process queue either grows until the heap is gone or blocks the caller when it fills, and everything still sitting in it dies with the process on precisely the crash you most wanted to read about. Getting the write off the request path is half the fix; getting it out of the process is the other half.
What the new pieces do
- Log Storedata
- Where parsed, sampled logs land durably — this is what a separate log-search system indexes and queries.
Step 2 · Decouple with a local agent
The app writes locally, an agent forwards
Fix Step 1: the application shouldn't talk to the network log path directly at all.
Introduce a local Log Agent — a sidecar or daemon running alongside the application. The app writes to a local, fast destination (stdout, a local file, a local socket) and returns immediately; the agent independently tails that output and forwards it asynchronously. The application's request path is now completely decoupled from the health of anything downstream.
Why this piece earns its place
This step quietly assumes the local write always succeeds and stays cheap, and the thing that breaks that assumption is the disk underneath it. The application writes to a file on the same volume it uses for everything else; if the agent dies, falls behind, or loses its ability to forward, that file grows until the volume is full — and logging has taken the application down again, just by a slower route. So ownership of that file is part of the design: rotation with a hard cap on total size, enforced by whoever writes it rather than by a cron job that may or may not run, and a truncate behaviour the agent tolerates without re-reading from the top. The agent’s read position is the other half. It has to be checkpointed to disk rather than held in memory, or a restart either replays everything already forwarded or skips whatever arrived while it was down — a duplicate flood or a silent gap, chosen by accident rather than by policy.
What the new pieces do
- Log Agentgateway
- A local sidecar/daemon that tails application output and forwards it asynchronously — the thing that decouples app latency from the health of the logging pipeline.
Step 3 · Backpressure and bounded buffers
What happens when downstream can't keep up?
The agent forwards to a durable queue. That queue — or whatever's past it — gets slow or falls behind. The agent's own local buffer starts filling. What should happen when it fills completely?
The agent's local buffer is full because downstream is slow. What's the right policy?
This recreates exactly the Step 1 problem — the application is now blocked on the logging pipeline's health again, just one hop further downstream. Blocking defeats the entire point of the local agent.
An unbounded buffer during a sustained downstream outage will eventually exhaust the host's memory or disk, taking down the AGENT (and potentially the host it runs on) — an unbounded buffer just delays and worsens the failure, not prevents it.
A bounded buffer with a clear, deliberate policy (spill to disk, then drop-oldest under sustained pressure) accepts that during a severe or prolonged downstream outage, SOME log volume may be lost — but the application itself is never blocked, and the host never runs out of resources. This is an explicit, acceptable trade-off, not an accident.
Give the agent a bounded, disk-backed buffer with an explicit overflow policy: spill from memory to local disk under pressure, and if the buffer is still full after that, drop the oldest (or lowest-priority) entries rather than blocking the application or growing without limit. Some log loss during a severe, sustained outage is an acceptable, deliberate trade — an application outage caused by its own logging pipeline is not.
Why this piece earns its place
The knob here is the buffer’s size, and the useful unit is time, not bytes: how long a downstream outage should the agent ride out without losing anything? Multiply the service’s steady log rate by that window and you have your number. Sizing by a round figure of megabytes instead means the quietest service survives for days and the noisiest for well under a minute on the same setting, which is how one number ends up wrong everywhere. Too small and an ordinary queue deploy costs you logs; too large and the spill file is competing for the same disk and the same IO the application needs. The direction of the drop deserves its own argument, because oldest-first sounds like the careless choice and is in fact the right one — the buffer only fills when something is going wrong, and the lines describing what is going wrong now are the ones arriving now. Dropping newest to preserve the backlog would keep the calmest minutes and discard the only ones anyone is going to read.
- boundedbuffer, not unlimited
- disk-spillbefore dropping anything
- drop-oldestlast resort, never block the app
What the new pieces do
- Durable Queueevents
- A durable buffer (think Kafka) absorbing bursts and outages downstream — bounded, with an explicit policy for what happens when it fills.
Step 4 · Batch for throughput
Don't send one network call per line
Forwarding every single log line as its own network write to the durable queue is enormously wasteful of connection overhead at real log volume. What's the fix?
Batch log lines — accumulate either N lines or T milliseconds of output, whichever comes first, then send one batched write. This amortizes network and write overhead across many lines, at the cost of a small, bounded delay (milliseconds to a couple seconds) before a given log line is actually durable downstream — an easy trade for the throughput gain at real scale.
Why this piece earns its place
Both halves of that trigger earn their place, and dropping either one fails in the opposite direction. Flushing on count alone strands a quiet service’s lines in the buffer indefinitely — nothing is lost, it simply never arrives, which during a low-traffic incident amounts to the same thing. Flushing on time alone throws away the efficiency on exactly the busy services that needed it. So set the timer by what someone tailing logs mid-incident will tolerate before they stop trusting the screen, set the count by the payload size the queue is happiest accepting, and let whichever fires first win. Going long on either costs more than latency: the batch is the unit of loss if the host dies before a flush, and it is also the unit of retry, so a larger batch means resending a lot of good lines to recover one bad send. That is also why the delivery key that comes next is per batch — the batch, not the line, is the thing this pipeline actually moves.
Step 5 · Where does parsing happen?
Raw text vs. structured fields
A raw log line is just a string. Downstream search, alerting, and dashboards need structured fields — timestamp, level, service name, trace id. Should parsing happen at the agent (distributed, at the edge) or centrally, after the queue?
Turning raw text into structured fields — parse at the edge (agent) or centrally, after the queue?
Edge parsing distributes CPU cost nicely, but with potentially thousands of independent agent instances each running parsing logic, small drift or bugs in parsing rules across different service versions/deployments become genuinely hard to keep consistent and hard to fix centrally.
Centralizing parsing means one codebase, one place to add a new log format or fix a parsing bug, and one thing to monitor for parsing errors — the trade-off is that this component now has to scale with the FULL aggregate log volume, which is a real, but very solvable (horizontally scalable) capacity planning problem.
Unparsed raw text makes downstream search and alerting radically slower and less capable (full-text scans instead of indexed field queries) — structuring at ingest time is what makes the rest of the observability stack fast.
Centralize structured parsing in a dedicated Structured Parser service reading from the durable queue — one place to define and evolve parsing rules across every log format, at the cost of a component that must scale horizontally with total log volume (a well-understood, solvable scaling problem, unlike inconsistency across thousands of independent edge parsers).
Why this piece earns its place
Consistency cuts both ways: the cost of having one place to fix a rule is that it is also one place to break every service at once. A bad rule on an edge agent corrupts the logs of whatever it runs beside; the same rule shipped to a central parser corrupts the whole fleet’s logs in the same minute, and the incident it obscures will be the one you are trying to read through. That makes two things non-negotiable here. The raw line travels with the structured fields, so a bad rule is recoverable by reparsing rather than a permanent loss of the original text. And a line that fails to parse is never dropped — it goes to the dead-letter path, so an unanticipated format shows up as a queue somebody can look at instead of as a service that appears to have gone quiet. Beyond that, rules get versioned and canaried against a slice of the live stream, because the cheapest place to find out a rule is wrong is on traffic nothing is depending on yet.
What the new pieces do
- Structured Parserservice
- Turns raw log lines into structured fields (level, timestamp, service, trace id) so downstream search and alerting are fast.
Step 6 · Delivery guarantees
At-least-once, plus dedup
A batch fails to deliver — a network blip mid-send. The agent retries. What if the batch had actually landed the first time, and the retry creates a duplicate?
A delivery might succeed even though the sender thinks it failed and retries. How do you avoid double-ingesting?
Silently losing logs on any transient network hiccup is a much worse trade than occasionally handling a duplicate — logs are often needed precisely to debug the incident that caused the network hiccup in the first place.
At-least-once delivery (retry until acknowledged) guarantees nothing is silently dropped due to a transient failure, and pairing it with an idempotency key (a deterministic id per batch) lets the downstream store recognize "I've already ingested this exact batch" and discard the duplicate rather than double-counting it.
True exactly-once delivery across independent network hops is a notoriously hard distributed-systems problem with real coordination overhead — at-least-once plus idempotent dedup achieves the same practical outcome (no duplicates ingested) far more simply.
Use at-least-once delivery — retry on any uncertain outcome — paired with a deterministic idempotency key per batch, so the downstream path (via the Dead-Letter/Retry store for anything that needed a retry) can recognize and discard a true duplicate rather than double-ingesting it. This achieves effectively-once ingestion without needing distributed transactions.
Why this piece earns its place
Both halves of this have a way of failing that nothing reports. The key has to be sealed with the batch and persisted alongside it, not computed at send time from whatever the batch happens to contain — an agent that restarts mid-retry and re-forms the batch from its buffer will sweep in a few lines that arrived meanwhile, hash to a different key, and the store dedups nothing. The retry lands as a perfectly clean duplicate wearing a fresh id. So a batch gets its identity once, when it is closed, and carries it through every attempt. The other half is the window: no store remembers every key it has ever seen, it remembers recent ones, and a retry that finally succeeds after a long outage can fall outside that memory and be ingested twice. The dedup retention therefore has to outlast the longest backoff the agent is permitted to take — which means the retry policy and the store’s retention are one decision made in two places, and they drift the day somebody tunes the backoff on its own.
What the new pieces do
- Dead-Letter / Retrydata
- Holds records that failed to parse or deliver on the first attempt, retried on a backoff instead of being silently dropped.
Step 7 · Sampling at high volume
You don't need 100% of everything
A fleet of thousands of instances emitting verbose debug-level logs can produce a genuinely enormous, expensive volume — most of it routine and never looked at. Should the pipeline ingest all of it?
Ingesting 100% of debug/info-level logs at fleet scale is expensive and most is never read. What's the fix?
Errors and warnings are exactly the events you can't afford to miss, so they're never sampled away. Routine, high-volume, low-signal logs are the right target for sampling — even keeping 1-5% of them still gives you a statistically useful picture of normal behavior at a fraction of the ingestion cost.
Disabling routine logging entirely removes a genuinely useful signal (the normal-behavior baseline you compare against when something goes wrong) — sampling keeps a representative slice instead of throwing the whole category away.
Compression helps but doesn't change the fundamental volume being processed, indexed, and stored — at extreme scale, the ingestion and indexing cost of the raw event COUNT, not just its compressed size, is often the dominant expense.
Introduce a Sampler: always keep 100% of high-signal logs (errors, warnings, anything tagged critical), but sample a smaller, representative fraction of high-volume, low-signal logs (routine debug/info). This preserves the ability to debug real incidents fully while controlling the cost of ingesting the much larger volume of routine noise.
Why this piece earns its place
Sample the wrong unit and you hold the volume down while destroying the thing logs are for. A per-line coin flip leaves every request holding a scattered handful of its own lines, so there is plenty of data and not one complete story to follow. The decision has to be made per request — hash the trace id and let that choose — so a request that is kept keeps all of its lines and a request that is dropped contributes none. It is also worth being honest about where the saving lands. The Sampler sits after the queue, so the agent, the network and the queue still carry the full flood; what gets cheaper is the store and everything indexing behind it. That is a real and usually dominant cost, but it means sampling is not a capacity control for the pipeline itself, and reaching for a lower sampling rate because the queue is struggling is treating the wrong component. The pipeline needs its own limit, further upstream.
What the new pieces do
- Samplerservice
- Keeps all high-signal logs (errors) and samples the flood of routine, low-signal ones — because ingesting 100% of everything is often prohibitively expensive at real scale.
Step 8 · The sharp edge
The logging storm
An incident hits: a retry loop somewhere starts logging an error on every failed attempt, at high frequency, across many instances simultaneously. Log volume spikes 100x in seconds — right when you most need the pipeline to be reliable and fast.
Add per-service rate limiting on the logging path itself, distinct from the sampling policy: if a single service's log volume spikes far beyond its normal baseline, cap it (drop excess volume beyond the cap, optionally keeping a representative slice) rather than letting it saturate the agent's buffer, the queue, and potentially the app itself via the Step 3 backpressure policy. This is the final application of the Step 1 principle: the observability system must never amplify the very incident it exists to help you debug.
Why this piece earns its place
A baseline-relative limit has a cold start, and the day it bites is a deploy. A service that has just shipped has no established normal, so it either runs unthrottled — precisely the wrong exemption for code that has never run in production before — or it inherits a default that throttles a legitimately chatty new service on its first day. The honest answer is to hold new services on a conservative default and let the baseline earn them a higher one over a settling period. Scope is the other trap: the limiter lives at the agent, which sees one instance, while the storms worth fearing are fleetwide, and a crash loop can trip no single instance’s cap at all while the fleet multiplies it by however many instances are running. A per-instance cap needs a fleetwide view behind it. And whatever the limiter drops has to be counted and reported as its own signal, because on a log-volume graph a limiter working perfectly looks identical to a service that has gone silent — and those two readings send on-call in opposite directions.
You did it
You just designed a logging pipeline.
Everything you assembled, in order
- Synchronous, direct writes couple application health to log-server health — the pipeline must never become a dependency the app blocks on.
- A local agent decouples the app from the network path entirely, buffering and forwarding asynchronously.
- A bounded, disk-backed buffer with an explicit drop-oldest overflow policy prevents the agent from either blocking the app or exhausting host resources.
- Batching amortizes write overhead across many log lines for a small, bounded delivery-latency cost.
- Structured parsing is centralized for consistency, at the cost of a component that must scale with total log volume.
- At-least-once delivery plus an idempotency key achieves effectively-once ingestion without distributed-transaction overhead.
- Sampling keeps 100% of high-signal logs (errors) and only a representative fraction of high-volume, low-signal ones.
- Per-service rate limiting on the logging path itself prevents a logging storm from turning an incident into an outage caused by its own observability tooling.