A written version of the interactive roadmap above — every station, what you'll learn, and a small thing to build — laid out for reading, reference and search.
Foundations Start here
F1. SQL, Deeply
Beginner · 75 min
SQL is the mother tongue of data engineering — you will write it every day. Go past SELECT: window functions, CTEs, joins at scale, aggregation, and reading a query plan. The difference between a junior and a senior data engineer is often just how well they know SQL.
Skills: Window functions & CTEs · Joins at scale · Aggregation & grouping · Query plans & EXPLAIN
Build it: Compute a 7-day rolling average per user with a window function, then check the query plan. Where would an index change the cost?
F2. Python for Data
Beginner · 60 min
Python is the glue of the data stack. Learn the data-wrangling toolkit — pandas (and its successors), working with APIs and files, and writing clean, testable transformation code. Most pipelines are Python calling SQL calling Python; be fluent in both.
Skills: pandas / dataframes · Reading files & APIs · Vectorized ops · Testable transforms
Build it: Load a messy CSV, clean the types and nulls, and reshape it wide-to-long. Which steps belong in SQL instead of pandas — and why?
F3. Data Modeling & Warehousing
Intermediate · 65 min
A warehouse is only as good as its model. Learn dimensional modeling — facts and dimensions, star schemas, slowly-changing dimensions — and how OLAP columnar stores differ from OLTP. Good modeling is what makes analytics fast and queries obvious.
Skills: Facts & dimensions · Star / snowflake schemas · Slowly-changing dimensions · OLAP vs OLTP
Build it: Model orders for analytics: what is the fact table, what are the dimensions, and how do you handle a customer who changes address?
F4. Batch vs Streaming
Intermediate · 60 min
The defining choice in data engineering: process data in scheduled batches, or continuously as it arrives. Learn the tradeoffs — latency, cost, complexity, correctness — and why most real platforms run both. This decision shapes every tool you pick downstream.
Skills: Batch processing · Stream processing · Latency vs cost · Lambda / Kappa architectures
Build it: A dashboard needs "yesterday" and a fraud check needs "now". Choose batch vs streaming for each and justify by latency and cost.
F5. File & Table Formats
Intermediate · 55 min
How data sits on disk decides how fast and cheap it is to query. Learn columnar formats (Parquet), compression, and open table formats (Iceberg, Delta) that bring ACID and time-travel to the data lake. Row vs column is the single biggest analytics performance lever.
Skills: Row vs columnar (Parquet) · Compression & encoding · Open table formats (Iceberg/Delta) · Schema evolution
Build it: A query scans one column of a billion-row table. Show why Parquet reads far less than CSV — and what predicate pushdown buys you.
F6. Orchestration Basics
Intermediate · 50 min
Pipelines are graphs of dependent tasks that must run in order, on schedule, and recover from failure. Learn the DAG model, scheduling, retries, backfills, and idempotency — the concepts that turn a pile of scripts into a reliable data platform.
Skills: DAGs & dependencies · Scheduling & retries · Backfills · Idempotent tasks
Build it: Task B depends on A, which sometimes fails. Design the DAG so a retry of A does not double-count, and a backfill is safe to re-run.
Core Data The craft
T1. ETL vs ELT
Intermediate · 55 min
Do you transform data before loading it (ETL) or load it raw and transform in the warehouse (ELT)? The modern cloud stack flipped to ELT — cheap storage and powerful warehouses changed the economics. Learn both, and why the answer depends on scale and tooling.
Skills: ETL vs ELT tradeoffs · Raw / staging / mart layers · Ingestion patterns · Idempotent loads
Build it: You are handed 20 source tables. Design the raw→staging→mart flow, and decide what transforms happen where — and why ELT wins here.
T2. Apache Spark
Advanced · 75 min
When data outgrows one machine, Spark spreads the work across a cluster. Learn the execution model — partitions, transformations vs actions, lazy evaluation, and the dreaded shuffle. Most big-data batch jobs run on Spark; understanding its cost model is essential.
Skills: Partitions & parallelism · Transformations vs actions · The shuffle · Caching & spilling
Build it: A join is painfully slow. Explain how a shuffle causes it, and two ways — broadcast, repartition — to make it fast.
T3. Kafka & Streaming
Advanced · 70 min
Streaming platforms move events continuously between systems. Learn Kafka — topics, partitions, consumer groups, offsets — and stream processing concepts: windowing, watermarks, exactly-once, and event time vs processing time. This is the backbone of real-time data.
Skills: Topics, partitions, offsets · Consumer groups · Windowing & watermarks · Event time vs processing time
Build it: Count events per 5-minute window, but data arrives late. Show how watermarks let you emit results without losing the stragglers.
T4. Airflow & Orchestration
Advanced · 65 min
Orchestrators like Airflow (and Dagster, Prefect) run your DAGs in production — scheduling, monitoring, retrying, and alerting on hundreds of pipelines. Learn operators, sensors, backfills, and dependency management. This is where the whole platform is coordinated.
Skills: DAG authoring · Operators & sensors · Backfills & catchup · Alerting on failures
Build it: A daily pipeline missed 3 days. Trigger a controlled backfill for exactly those dates without re-running everything downstream.
T5. dbt & Transformations
Advanced · 60 min
dbt brought software engineering to analytics: transformations as version-controlled SQL models, with tests, docs, lineage, and dependency-aware runs. Learn models, refs, tests, and materializations. It is how modern teams keep a warehouse trustworthy and maintainable.
Skills: Models & refs · Tests & assertions · Materializations · Lineage & docs
Build it: Turn a 300-line SQL script into layered dbt models with tests. How does ref() give you a dependency graph and safe rebuilds?
T6. Data Lakes & Lakehouse
Advanced · 60 min
The lakehouse merges the cheap, flexible storage of a data lake with the reliability of a warehouse. Learn the architecture — object storage plus open table formats plus a query engine — and the medallion (bronze/silver/gold) layering that organizes raw to refined.
Skills: Lake vs warehouse vs lakehouse · Object storage + table formats · Medallion layering · Query engines (Trino/Spark)
Build it: Design a lakehouse for clickstream data: where do raw events land, how do they become clean tables, and what queries the gold layer?
Production Operate & scale
P1. Data Quality & Testing
Advanced · 60 min
Bad data is worse than no data — it erodes trust silently. Learn data quality dimensions (completeness, freshness, uniqueness), assertion tests, anomaly detection, and contracts between producers and consumers. A pipeline that runs green but ships wrong numbers has failed.
Skills: Quality dimensions · Assertion & schema tests · Anomaly detection · Data contracts
Build it: Overnight a source silently dropped 30% of rows. Which test catches it before the dashboard does — and how do you alert, not just log?
P2. Governance & Lineage
Advanced · 55 min
At scale, you must know where data came from, who can see it, and what depends on it. Learn lineage (column-level provenance), cataloging, access control, and PII handling. Governance is what lets a big org trust and safely share its data.
Skills: Column-level lineage · Data catalog · Access control & PII · Compliance (GDPR)
Build it: A column must be deleted for GDPR. Use lineage to find every downstream table and dashboard it touches before you drop it.
P3. Observability & SLAs
Advanced · 55 min
Data platforms need the same observability as services — plus data-specific signals: freshness, volume, and schema drift. Learn pipeline monitoring, SLAs on when tables update, and how to page on a stale mart before the business notices.
Skills: Freshness / volume / schema monitors · Pipeline metrics · Data SLAs · Alerting on staleness
Build it: A downstream mart must be ready by 8am. Set an SLA and the freshness monitor that pages if the upstream job runs late.
P4. Partitioning & Performance
Advanced · 65 min
Big tables demand smart physical layout. Learn partitioning, clustering, file sizing (the small-files problem), and how partition pruning turns a full scan into a targeted read. This is where query cost and speed are won or lost at the storage layer.
Skills: Partitioning & clustering · Partition pruning · Small-files problem · Compaction
Build it: A daily query scans a year of data to read one day. Partition the table so it prunes to a single day — and fix the tiny-files fallout.
P5. Cost & Scaling
Advanced · 50 min
Cloud data platforms bill for compute and scanned bytes, and costs sneak up fast. Learn warehouse sizing, query cost attribution, incremental processing, and caching. The senior skill is delivering the same numbers for a fraction of the spend.
Skills: Compute vs storage billing · Bytes-scanned optimization · Incremental processing · Cost attribution
Build it: A dbt job re-processes all history nightly. Make it incremental so it touches only new data — and estimate the cost you just saved.
P6. Real-time & CDC
Advanced · 60 min
Change Data Capture streams every insert/update/delete from a source database into your platform in near real time, without heavy batch pulls. Learn CDC (log-based), streaming upserts, and materializing real-time views. This is how modern platforms stay fresh to the second.
Skills: Change Data Capture (CDC) · Log-based replication · Streaming upserts · Real-time materialized views
Build it: Mirror a production Postgres table into the warehouse within seconds of each write. Why is log-based CDC better than polling for changes?