The whole design, in writing
Learn system design by building a log aggregation and search system like the ELK stack (Elasticsearch, Logstash, Kibana) or Loki step by step. An interactive guide covering why grep-over-SSH can't scale, shipping logs with agents, a Kafka buffer that absorbs bursts, a parse/enrich pipeline, the inverted index that powers full-text search, time-based sharded indices with scatter-gather queries, hot-warm-cold retention, and the unhappy paths (mapping explosion, backpressure, PII).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is a log search system?
During an incident you need to find, in seconds, "every ERROR from the payments service in eu-west mentioning timeout in the last 20 minutes" — across thousands of hosts producing billions of lines a day. SSHing into boxes to grep can't find logs from a host that already scaled down, and can't search the fleet at once.
A log search system (ELK/Loki) ships every log line to a central place, structures it, and builds an inverted index so you can full-text search the entire fleet's history in milliseconds. Collect everywhere → buffer → parse → index → search. It turns scattered text files into one queryable, durable record.
What the new pieces do
- Log Sourcesclient
- Every app, container and host writing log lines — a firehose of unstructured and structured events from across the fleet.
Step 1 · Why not just grep?
SSH-and-grep doesn't scale
The starting point: logs are files on each server; when you need something, SSH in and grep. Fine for a few boxes. Why is it hopeless for a real fleet?
grep over SSH across the fleet. Why does it fail?
grep itself is fast on one file. The failure is structural: you can't grep a host that's gone, can't search thousands of boxes at once, and logs vanish when ephemeral hosts die.
Local disk pressure is a symptom. The core problems are no central search, no durability past a host's lifetime, and no way to query the whole fleet together.
Ephemeral hosts scale down and take their logs with them; you can't fan a grep across thousands of boxes in an incident; and there's no unified, retained, searchable history. Centralizing and indexing fixes all three.
Local grep fails structurally: logs die with ephemeral hosts, you can't search the whole fleet at once, and there's no durable, unified history. The fix is to ship every line to a central system that keeps it and indexes it — so search is fleet-wide, instant, and survives the host.
Why this piece earns its place
Worth saying what centralizing costs, because it isn’t free: it turns per-host, independently-failing text files into one shared dependency that every team leans on at the same moment. The day the pipeline is behind or down, nobody can see anything — and it tends to be behind for the same reason everything else is, since the incident that makes you need logs is also the incident producing the flood. So I’d keep the local files. Hosts still write and rotate logs on disk with a short local window, purely as the escape hatch: when the search layer is unavailable, whoever is on call can get onto a box that’s still alive and read the raw thing. The corollary is that the pipeline must not observe itself — shippers, buffer and indexer report their own health through metrics and a separate path, not by writing into the index they’re responsible for filling. Otherwise the one outage you can’t diagnose is an outage in the system you built to diagnose outages.
Step 2 · Make logs searchable at the source
Structure the logs
A raw line like "payment failed for user 42 after 3011ms" is human-readable but hard to query precisely ("all payments slower than 3s in eu"). What should a log be so it's searchable?
How should log lines be shaped for precise search?
Pure free text forces fragile regex parsing downstream and makes structured queries (latency > 3s, service=payments) hard. Emitting structured fields at the source is far more reliable and queryable.
Structured logging attaches typed fields (level=ERROR, service=payments, latency_ms=3011, region=eu) so queries filter and aggregate precisely, and full-text still works on the message. Late-parsing unstructured logs is a fallback, not the goal.
A single blob field can be full-text searched but not filtered or aggregated by attribute. You want both: structured fields for filtering plus a message for full-text.
Emit structured logs — typically JSON with fields like level, service, region, latency_ms, trace_id, and a free-text message. Structured fields make precise filtering and aggregation possible ("service=payments AND latency_ms>3000"); the message keeps full-text search. Where you can't control the format, the pipeline parses lines into fields on ingest instead.
Why this piece earns its place
Structured logging quietly assumes you own the code that emits. A good share of the fleet’s lines come from things that will never emit your JSON — the proxy, the database, the runtime, the kernel, a vendor’s sidecar — so the parse path downstream isn’t a fallback you might need, it’s permanent, and you end up maintaining two routes to the same document shape and keeping them in agreement. The thing I’d settle before anyone writes a logger is the field names. A field name is a public interface: every saved query, alert and dashboard is written against it. Renaming service to svc later doesn’t fail loudly, it just stops matching — and because old indices keep the old name until they age out, every query has to speak both names for the whole retention window. So agree one shared schema up front for host, service, region, severity and trace id, and treat a change to it as a migration with a cutover, not a commit.
Step 3 · Get logs off the box
Shipper agents
Logs are written to files on thousands of short-lived hosts. Something has to reliably read them and forward them out — before the host disappears — without the app having to care.
Run a lightweight shipper agent (Filebeat, Fluent Bit, Vector) on every host or as a sidecar. It tails log files, remembers its read position (so a restart doesn't re-send or lose lines), adds metadata (host, container, service, region), and forwards batches onward. The app just writes logs; the agent handles reliable delivery off the box.
Why this piece earns its place
The agent is now the most widely deployed program you own — it runs on every host, so its bugs are fleet-wide, and its hardest hour is an incident hour, when the box is already struggling and log volume is spiking. That makes it a plausible cause of the next outage, not just a witness to it. So I’d give it a hard resource ceiling and make it degrade by falling behind and dropping, never by blocking the application writing the file. The race worth naming is rotation: a checkpoint only helps if the file is still there. If the app rotates or truncates faster than a backed-up agent reads, lines are collected by the rotation before the tail reaches them, and nothing anywhere reports an error. So local file retention has to be sized against the agent’s worst-case backlog. And the agent has to prove it is alive rather than be assumed alive: a tail that stops raises nothing, and a silent host is indistinguishable from one whose service simply had a quiet hour. You can’t read an agent’s health off the lines it sends, because its failure looks exactly like having nothing to send. So it heartbeats on the separate channel step 1 reserved for the pipeline’s own telemetry, carrying its read offset, and the alarm fires on a heartbeat that stopped rather than on someone eventually noticing absent logs.
What the new pieces do
- Shipper Agentgateway
- A lightweight agent (Filebeat/Fluentd) on each host that tails log files, adds metadata (host, service), and forwards them out.
Step 4 · Absorb the flood
A durable buffer
Log volume is spiky — and it spikes hardest during an incident, exactly when the indexer is already stressed and you most need the data. If shippers write straight to the indexer, a burst overruns it and you lose logs. How do you smooth it?
Log volume spikes during incidents. How do you avoid overrunning the indexer?
Sizing for peak is wasteful most of the time and still fails on an unexpected spike. A buffer lets you size the indexer for sustained throughput while a queue absorbs bursts.
Synchronous retry pushes backpressure onto the hosts and can stall them; under a big burst you still drop. A durable queue between shippers and indexer decouples their speeds cleanly.
Shippers write to Kafka, which durably holds the burst; the indexer consumes at its own sustainable rate. Bursty producers and a steady consumer are decoupled, so a spike queues rather than drops, and you can replay after an indexer outage.
Insert a durable buffer (Kafka or similar) between the shippers and the indexer. Shippers write logs to the buffer; the indexer consumes at its own steady rate. A burst is absorbed by the queue instead of overrunning the indexer, an indexer outage just makes the backlog grow (then drain), and you can replay the log if you need to re-index. It's the shock absorber of the pipeline.
Why this piece earns its place
The buffer introduces one number that decides how much of an outage you survive: how long it retains. That retention is exactly the window the indexer can be dead before data is gone for good. Set it in hours and a failure starting on a Friday evening eats the weekend; set it in weeks and you’re paying to hold a second full copy of every line on the queue’s own disks. I’d size it against a realistic worst repair time rather than the average one, and alarm on consumer lag rather than queue depth, since lag is the thing that converts into loss. The second consequence is subtler, and it bites during precisely the incident the buffer was bought for: a queue turns dropped logs into late logs. As the backlog grows, lines land in the index minutes after they happened, and an engineer staring at search concludes the code path never ran when it simply hasn’t been indexed yet. So surface current ingestion lag in the UI — a search over a window that is still filling should say so.
- burstabsorbed, not dropped
- replayre-index from the log
- decoupledproducer ≠ consumer speed
What the new pieces do
- Bufferbus
- A durable queue that absorbs bursts and decouples bursty shippers from the slower indexer. Logs wait here safely under load.
Step 5 · Parse & enrich
The ingest pipeline
Between the buffer and the index, raw lines need to become clean, typed, enriched documents — parsed, fields extracted, timestamps normalized, sensitive data handled — consistently, at high throughput.
Run an ingest pipeline (Logstash/Vector/ingest nodes) that consumes from the buffer and, per line: parses unstructured text into fields (grok/regex/JSON), normalizes the timestamp and types, enriches (geo-IP, service tags), optionally redacts PII, and drops or samples noisy logs. It emits clean documents to the indexer. Keep it stateless and horizontally scalable so it scales with volume.
Why this piece earns its place
Parsing is the one stage whose cost per line isn’t bounded by anything you control. A regex over a log line is cheap until the line is a stack trace and the pattern behind it backtracks, and a single badly-written expression on a high-volume service is enough to collapse throughput for every other service sharing the pipeline — a noisy-neighbour problem where the neighbour is a pattern nobody reviewed. So anchor patterns, put a time bound on per-line parse work, and keep pushing format ownership back to the emitters, because a line that arrives already structured skips this risk entirely. The other thing to build here is somewhere for failures to go. A line matching no pattern must not be dropped silently and must not be retried forever: it goes to a dead-letter index with its raw text preserved and a counter on it. That counter is one of the more useful alarms on the whole system — it jumps the day a service changes its log format, which is the day the dashboards built on those fields would otherwise quietly go empty.
What the new pieces do
- Ingest Pipelineservice
- Parses raw lines into structured fields, enriches them, and builds the inverted index that makes full-text search fast.
Step 6 · Search a trillion lines
Inverted index + sharding
Now the core question: how do you full-text search billions of documents in milliseconds, when a linear scan would take hours? And how does that scale past one machine?
How do you full-text search billions of docs in milliseconds?
A linear scan of billions of docs per query is hopeless. Search engines pre-build an inverted index (term → list of documents containing it) so a query is a fast index lookup, not a scan.
For each term, store the list of documents (and positions) that contain it; a query intersects/unions posting lists to find matches instantly. Documents are split into shards across machines; a query fans out to all shards in parallel and merges the ranked results.
LIKE with a leading wildcard can't use a normal index and scans the table — far too slow at this scale. Full-text search needs a purpose-built inverted index, not a relational substring match.
Build an inverted index: for every term, keep a posting list of the documents (and positions) containing it, so a query becomes fast set operations over posting lists instead of a scan. Shard the documents across machines (usually time-based indices — one per day) and query scatter-gather: fan the query to all relevant shards in parallel, then merge and rank the hits. Replicate shards for availability. This is what Lucene/Elasticsearch do.
Why this piece earns its place
The knob this introduces is shard size, and both directions hurt. A shard is a whole index with fixed overhead, so cutting the data finely — one per service per day — makes a query pay fan-out and merge cost across hundreds of tiny shards until coordination dominates the actual searching. Go the other way and a shard grows past what one machine can rebuild, relocate or recover in a sensible time, and you lose the ability to rebalance at all. I’d pick a target size per shard and roll over on size as well as on the calendar, so a quiet day and a flood day both produce shards in that range. And I’d be precise about what scatter-gather does to latency: a query finishes when its slowest shard finishes, so the tail of the per-shard distribution is the user-visible number, not the average — one degraded node makes every query that touches it slow. That argues for per-shard timeouts and returning partial results clearly marked as partial, rather than making every search wait on the worst machine.
- inverted indexterm → docs
- time-based shardsskip out-of-range days
- scatter-gatherparallel + merge
What the new pieces do
- Index Shard Adata
- A shard of the time-based index: an inverted index (term → documents) plus stored fields, replicated for durability.
- Index Shard Bdata
- Another shard. Documents are spread across shards so both indexing and search scale horizontally.
- Search UIui
- The query layer (Kibana): fans a query to all relevant shards, merges the ranked hits, and renders results and dashboards.
Step 7 · Don't keep everything forever
Retention & hot-warm-cold tiers
Logs are enormous and their value drops fast — you query today's constantly, last month's rarely, last year's almost never. Keeping it all on fast storage is ruinous. How do you bound cost without losing recent speed?
Use time-based indices with rollover and a hot-warm-cold lifecycle: fresh indices live on fast SSD hot nodes (heavy indexing + querying); as they age they move to warm (cheaper, less-queried) and cold/frozen (cheap object storage, rarely queried) tiers; past retention they're deleted or archived. Because each day is a separate index, aging and deletion are just moving or dropping whole indices.
Why this piece earns its place
The whole lifecycle assumes queries stay near the present, and most do. What breaks it is the query that doesn’t: a year-wide dashboard, or a compliance request for one identifier across everything you still hold. That query fans out to tiers chosen for cheapness, and the damage isn’t only that it’s slow — dragging cold data back through the cluster evicts the caches and eats the memory the hot path depends on, so one person’s historical question degrades everybody’s incident. I’d bound it deliberately: a default time window in the UI, an explicit ceiling on how far back an interactive search may reach, and long-range work run against the cold tier in a way that can’t contend with hot search. The other quiet assumption lives in the word deleted. Dropping an index past retention only honours a deletion promise if nothing else holds a copy, and snapshots routinely outlive the policy that supposedly expired them. Drive the backup lifecycle from the same clock, or the answer you give an auditor isn’t true.
- hotSSD · today
- warm/coldcheap · older
- deletepast retention
Step 8 · The sharp edges
Mapping explosion, backpressure & PII
Real log pipelines get bitten by: a field with unbounded distinct keys blowing up the index (mapping explosion), the indexer falling behind under sustained load, and sensitive data (passwords, tokens, PII) landing in searchable storage.
Prevent mapping explosion by capping fields and not indexing arbitrary user-supplied keys (or using schema-less stores like Loki that index only labels + full-text the body). Handle backpressure with the buffer, autoscaling indexers, and dropping/sampling low-value logs first. Redact PII in the ingest pipeline before indexing, and enforce access controls + retention for compliance. And separate index-heavy from search-heavy nodes so ingestion spikes don't slow queries.
Why this piece earns its place
Shedding is the only one of these guards that has to work while you’re distracted, so the decision can’t be made in the moment. If the rule is to drop whatever is loudest, the pipeline sheds the service producing the most lines — and during an incident that is very often the broken service whose logs are the reason anyone is looking. Volume is a terrible proxy for value. So I’d have the shipper stamp each stream with a declared class: audit and security lines that are never shed, operational lines that may be sampled, debug chatter that goes first. Then shedding executes a policy someone wrote calmly instead of a heuristic that punishes the noisy. Two things follow. The classes have to be agreed with the teams who own those logs rather than assigned by the platform alone, or everything gets declared critical and the policy means nothing. And the shed path has to be rehearsed, because a path that only ever runs during a real emergency is a path nobody has watched work.
You did it
You just designed a log search system.
Everything you assembled, in order
- grep-over-SSH fails: logs die with ephemeral hosts and you can't search the fleet at once.
- Emit structured logs (fields + message) so search can filter, aggregate and full-text.
- Shipper agents tail files, checkpoint offsets, enrich metadata, and forward off the host.
- A durable buffer (Kafka) absorbs bursts and decouples bursty shippers from the steady indexer.
- An ingest pipeline parses, normalizes, enriches and redacts before indexing.
- An inverted index + time-based sharding + scatter-gather search billions of lines in ms.
- Hot-warm-cold retention bounds cost; cap cardinality, redact PII, and split node roles.