Data & Retrieval

Batch Processing

Processing data in large scheduled chunks rather than continuously as it arrives.

Batch processing runs computations over a large, bounded set of data collected over a period, on a schedule — hourly, nightly, weekly. It optimizes for throughput and completeness over latency: you accept results are minutes-to-hours stale in exchange for simpler logic, easy reprocessing, and efficient bulk I/O.

Worked example: a nightly job reads the day’s full order table, computes revenue by region, and writes a report — simple to reason about because it sees the complete, unchanging dataset in one pass. Gotcha: batch is not obsolete — it is still the right choice when freshness does not matter and correctness does, and mixing it with streaming (as in lambda architecture) is often better than forcing everything real-time, since a nightly recompute is far simpler than exactly-once streaming state.