The big idea
What is a log search system?
During an incident you need to find, in seconds, "every ERROR from the payments service in eu-west mentioning timeout in the last 20 minutes" — across thousands of hosts producing billions of lines a day. SSHing into boxes to grep can't find logs from a host that already scaled down, and can't search the fleet at once.
A log search system (ELK/Loki) ships every log line to a central place, structures it, and builds an inverted index so you can full-text search the entire fleet's history in milliseconds. Collect everywhere → buffer → parse → index → search. It turns scattered text files into one queryable, durable record.
How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the pipeline grow, and at the end kill the buffer and lose a shard to see what keeps ingest and search alive under load. Hit Begin.
Step 1 · Why not just grep?
SSH-and-grep doesn't scale
The starting point: logs are files on each server; when you need something, SSH in and grep. Fine for a few boxes. Why is it hopeless for a real fleet?
Design decision: grep over SSH across the fleet. Why does it fail?
The call: No central search, no durability past a host's life, and you can't query thousands of hosts at once. — Ephemeral hosts scale down and take their logs with them; you can't fan a grep across thousands of boxes in an incident; and there's no unified, retained, searchable history. Centralizing and indexing fixes all three.
Local grep fails structurally: logs die with ephemeral hosts, you can't search the whole fleet at once, and there's no durable, unified history. The fix is to ship every line to a central system that keeps it and indexes it — so search is fleet-wide, instant, and survives the host.
Centralize + index: Two moves: centralize (get logs off the hosts into durable storage before the host disappears) and index (build a structure that makes search fast, not a linear scan). Everything below implements those two ideas at scale.
Step 2 · Make logs searchable at the source
Structure the logs
A raw line like "payment failed for user 42 after 3011ms" is human-readable but hard to query precisely ("all payments slower than 3s in eu"). What should a log be so it's searchable?
Design decision: How should log lines be shaped for precise search?
The call: Emit structured logs (JSON) with fields: level, service, latency, message, plus context. — Structured logging attaches typed fields (level=ERROR, service=payments, latency_ms=3011, region=eu) so queries filter and aggregate precisely, and full-text still works on the message. Late-parsing unstructured logs is a fallback, not the goal.
Emit structured logs — typically JSON with fields like level, service, region, latency_ms, trace_id, and a free-text message. Structured fields make precise filtering and aggregation possible ("service=payments AND latency_ms>3000"); the message keeps full-text search. Where you can't control the format, the pipeline parses lines into fields on ingest instead.
Structure early: A field you set at the source is reliable; a field you regex out of free text later is fragile. Structured logging (and correlation ids like trace_id) is what lets logs join up with traces and metrics — the whole observability story clicks together.
Step 3 · Get logs off the box
Shipper agents
Logs are written to files on thousands of short-lived hosts. Something has to reliably read them and forward them out — before the host disappears — without the app having to care.
Run a lightweight shipper agent (Filebeat, Fluent Bit, Vector) on every host or as a sidecar. It tails log files, remembers its read position (so a restart doesn't re-send or lose lines), adds metadata (host, container, service, region), and forwards batches onward. The app just writes logs; the agent handles reliable delivery off the box.
Tail + checkpoint: The agent decouples the app from the pipeline: apps write to stdout/files as normal, agents own buffering, retry, backpressure and metadata enrichment. Tracking file offset (a checkpoint) gives at-least-once delivery across restarts — you may re-send a few lines, but you won't lose them.
Step 4 · Absorb the flood
A durable buffer
Log volume is spiky — and it spikes hardest during an incident, exactly when the indexer is already stressed and you most need the data. If shippers write straight to the indexer, a burst overruns it and you lose logs. How do you smooth it?
Design decision: Log volume spikes during incidents. How do you avoid overrunning the indexer?
The call: Put a durable buffer (Kafka) between shippers and the indexer. — Shippers write to Kafka, which durably holds the burst; the indexer consumes at its own sustainable rate. Bursty producers and a steady consumer are decoupled, so a spike queues rather than drops, and you can replay after an indexer outage.
Insert a durable buffer (Kafka or similar) between the shippers and the indexer. Shippers write logs to the buffer; the indexer consumes at its own steady rate. A burst is absorbed by the queue instead of overrunning the indexer, an indexer outage just makes the backlog grow (then drain), and you can replay the log if you need to re-index. It's the shock absorber of the pipeline.
Decouple producers from consumers: Producers (thousands of bursty shippers) and the consumer (a rate-limited indexer) have mismatched speeds. A durable log queue lets each run at its own pace, turns spikes into backlog instead of loss, and adds replayability — the same pattern behind every robust ingestion pipeline.
Step 5 · Parse & enrich
The ingest pipeline
Between the buffer and the index, raw lines need to become clean, typed, enriched documents — parsed, fields extracted, timestamps normalized, sensitive data handled — consistently, at high throughput.
Run an ingest pipeline (Logstash/Vector/ingest nodes) that consumes from the buffer and, per line: parses unstructured text into fields (grok/regex/JSON), normalizes the timestamp and types, enriches (geo-IP, service tags), optionally redacts PII, and drops or samples noisy logs. It emits clean documents to the indexer. Keep it stateless and horizontally scalable so it scales with volume.
ETL for logs: This is extract-transform-load for telemetry. Doing parsing/enrichment/redaction here — once, centrally — beats every downstream query re-parsing raw text, and is where you enforce schema, cut noise, and protect sensitive fields before they're ever indexed.
Step 6 · Search a trillion lines
Inverted index + sharding
Now the core question: how do you full-text search billions of documents in milliseconds, when a linear scan would take hours? And how does that scale past one machine?
Design decision: How do you full-text search billions of docs in milliseconds?
The call: An inverted index (term → posting list of docs), sharded across machines and queried scatter-gather. — For each term, store the list of documents (and positions) that contain it; a query intersects/unions posting lists to find matches instantly. Documents are split into shards across machines; a query fans out to all shards in parallel and merges the ranked results.
Build an inverted index: for every term, keep a posting list of the documents (and positions) containing it, so a query becomes fast set operations over posting lists instead of a scan. Shard the documents across machines (usually time-based indices — one per day) and query scatter-gather: fan the query to all relevant shards in parallel, then merge and rank the hits. Replicate shards for availability. This is what Lucene/Elasticsearch do.
Inverted index + scatter-gather: The inverted index turns "find docs containing X" from an O(N)-scan into an O(matches) lookup. Sharding by time lets a query touch only the days in range and drop old indices whole. Scatter-gather runs each shard in parallel, so search stays fast as data grows — add shards, add throughput.
Step 7 · Don't keep everything forever
Retention & hot-warm-cold tiers
Logs are enormous and their value drops fast — you query today's constantly, last month's rarely, last year's almost never. Keeping it all on fast storage is ruinous. How do you bound cost without losing recent speed?
Use time-based indices with rollover and a hot-warm-cold lifecycle: fresh indices live on fast SSD hot nodes (heavy indexing + querying); as they age they move to warm (cheaper, less-queried) and cold/frozen (cheap object storage, rarely queried) tiers; past retention they're deleted or archived. Because each day is a separate index, aging and deletion are just moving or dropping whole indices.
Age = tier: Match storage cost to access frequency: recent = fast/expensive, old = slow/cheap, ancient = gone. Time-based indices make this trivial — a lifecycle policy rolls each index hot → warm → cold → delete on a schedule, keeping the fast tier small and search on recent data snappy.
Step 8 · The sharp edges
Mapping explosion, backpressure & PII
Real log pipelines get bitten by: a field with unbounded distinct keys blowing up the index (mapping explosion), the indexer falling behind under sustained load, and sensitive data (passwords, tokens, PII) landing in searchable storage.
Prevent mapping explosion by capping fields and not indexing arbitrary user-supplied keys (or using schema-less stores like Loki that index only labels + full-text the body). Handle backpressure with the buffer, autoscaling indexers, and dropping/sampling low-value logs first. Redact PII in the ingest pipeline before indexing, and enforce access controls + retention for compliance. And separate index-heavy from search-heavy nodes so ingestion spikes don't slow queries.
Design for the unhappy path: Exploding fields → cap/label-only indexing. Overrun → buffer + autoscale + drop noise. Secrets in logs → redact before index. Ingest vs search contention → dedicate node roles. The elegant index needs these guards to stay affordable, fast and compliant.
You did it
You just designed a log search system.
- grep-over-SSH fails: logs die with ephemeral hosts and you can't search the fleet at once.
- Emit structured logs (fields + message) so search can filter, aggregate and full-text.
- Shipper agents tail files, checkpoint offsets, enrich metadata, and forward off the host.
- A durable buffer (Kafka) absorbs bursts and decouples bursty shippers from the steady indexer.
- An ingest pipeline parses, normalizes, enriches and redacts before indexing.
- An inverted index + time-based sharding + scatter-gather search billions of lines in ms.
- Hot-warm-cold retention bounds cost; cap cardinality, redact PII, and split node roles.