Data
Engineer.

F1 · Beginner · 75 min

SQL is the mother tongue of data engineering — you will write it every day. Go past SELECT: window functions, CTEs, joins at scale, aggregation, and reading a query plan. The difference between a junior and a senior data engineer is often just how well they know SQL.

• Window functions & CTEs
• Joins at scale
• Aggregation & grouping
• Query plans & EXPLAIN

Build itCompute a 7-day rolling average per user with a window function, then check the query plan. Where would an index change the cost?

✓ CheckpointExplain what a window function does that GROUP BY cannot, using a running total as the example.

F2 · Beginner · 60 min

Python is the glue of the data stack. Learn the data-wrangling toolkit — pandas (and its successors), working with APIs and files, and writing clean, testable transformation code. Most pipelines are Python calling SQL calling Python; be fluent in both.

• pandas / dataframes
• Reading files & APIs
• Vectorized ops
• Testable transforms

Build itLoad a messy CSV, clean the types and nulls, and reshape it wide-to-long. Which steps belong in SQL instead of pandas — and why?

✓ CheckpointExplain when pandas stops being the right tool for a dataset, using a property of the work rather than a row count.

F3 · Intermediate · 65 min

A warehouse is only as good as its model. Learn dimensional modeling — facts and dimensions, star schemas, slowly-changing dimensions — and how OLAP columnar stores differ from OLTP. Good modeling is what makes analytics fast and queries obvious.

• Facts & dimensions
• Star / snowflake schemas
• Slowly-changing dimensions
• OLAP vs OLTP

Build itModel orders for analytics: what is the fact table, what are the dimensions, and how do you handle a customer who changes address?

✓ CheckpointExplain what a slowly-changing dimension is protecting, and what a report loses if you overwrite the row instead.

F4 · Intermediate · 60 min

The defining choice in data engineering: process data in scheduled batches, or continuously as it arrives. Learn the tradeoffs — latency, cost, complexity, correctness — and why most real platforms run both. This decision shapes every tool you pick downstream.

• Batch processing
• Stream processing
• Latency vs cost
• Lambda / Kappa architectures

Build itA dashboard needs "yesterday" and a fraud check needs "now". Choose batch vs streaming for each and justify by latency and cost.

✓ CheckpointAsk what happens if this result arrives an hour late instead of instantly. If the answer is “nothing bad”, defend batch.

F5 · Intermediate · 55 min

How data sits on disk decides how fast and cheap it is to query. Learn columnar formats (Parquet), compression, and open table formats (Iceberg, Delta) that bring ACID and time-travel to the data lake. Row vs column is the single biggest analytics performance lever.

• Row vs columnar (Parquet)
• Compression & encoding
• Open table formats (Iceberg/Delta)
• Schema evolution

Build itA query scans one column of a billion-row table. Show why Parquet reads far less than CSV — and what predicate pushdown buys you.

✓ CheckpointExplain why a columnar format wins on an aggregate over two columns and loses when you need the whole row.

F6 · Intermediate · 50 min

Pipelines are graphs of dependent tasks that must run in order, on schedule, and recover from failure. Learn the DAG model, scheduling, retries, backfills, and idempotency — the concepts that turn a pile of scripts into a reliable data platform.

• DAGs & dependencies
• Scheduling & retries
• Backfills
• Idempotent tasks

Build itTask B depends on A, which sometimes fails. Design the DAG so a retry of A does not double-count, and a backfill is safe to re-run.

✓ CheckpointExplain why a pipeline step must be idempotent before you are allowed to retry it automatically.

T1 · Intermediate · 55 min

Do you transform data before loading it (ETL) or load it raw and transform in the warehouse (ELT)? The modern cloud stack flipped to ELT — cheap storage and powerful warehouses changed the economics. Learn both, and why the answer depends on scale and tooling.

• ETL vs ELT tradeoffs
• Raw / staging / mart layers
• Ingestion patterns
• Idempotent loads

Build itYou are handed 20 source tables. Design the raw→staging→mart flow, and decide what transforms happen where — and why ELT wins here.

✓ CheckpointExplain what ELT gives you that ETL does not, and what you now owe the warehouse in return.

T2 · Advanced · 75 min

When data outgrows one machine, Spark spreads the work across a cluster. Learn the execution model — partitions, transformations vs actions, lazy evaluation, and the dreaded shuffle. Most big-data batch jobs run on Spark; understanding its cost model is essential.

• Partitions & parallelism
• Transformations vs actions
• The shuffle
• Caching & spilling

Build itA join is painfully slow. Explain how a shuffle causes it, and two ways — broadcast, repartition — to make it fast.

✓ CheckpointExplain what a shuffle costs in Spark and which operations cause one — this is where most jobs actually spend their time.

T3 · Advanced · 70 min

Streaming platforms move events continuously between systems. Learn Kafka — topics, partitions, consumer groups, offsets — and stream processing concepts: windowing, watermarks, exactly-once, and event time vs processing time. This is the backbone of real-time data.

• Topics, partitions, offsets
• Consumer groups
• Windowing & watermarks
• Event time vs processing time

Build itCount events per 5-minute window, but data arrives late. Show how watermarks let you emit results without losing the stragglers.

✓ CheckpointExplain what a watermark decides, and what happens to an event that arrives after it.

T4 · Advanced · 65 min

Orchestrators like Airflow (and Dagster, Prefect) run your DAGs in production — scheduling, monitoring, retrying, and alerting on hundreds of pipelines. Learn operators, sensors, backfills, and dependency management. This is where the whole platform is coordinated.

• DAG authoring
• Operators & sensors
• Backfills & catchup
• Alerting on failures

Build itA daily pipeline missed 3 days. Trigger a controlled backfill for exactly those dates without re-running everything downstream.

✓ CheckpointExplain why a backfill is the operation most likely to break a DAG that runs fine daily.

T5 · Advanced · 60 min

dbt brought software engineering to analytics: transformations as version-controlled SQL models, with tests, docs, lineage, and dependency-aware runs. Learn models, refs, tests, and materializations. It is how modern teams keep a warehouse trustworthy and maintainable.

• Models & refs
• Tests & assertions
• Materializations
• Lineage & docs

Build itTurn a 300-line SQL script into layered dbt models with tests. How does ref() give you a dependency graph and safe rebuilds?

✓ CheckpointExplain what a dbt test asserts that a unit test does not — the code was fine and the data still wrong.

T6 · Advanced · 60 min

The lakehouse merges the cheap, flexible storage of a data lake with the reliability of a warehouse. Learn the architecture — object storage plus open table formats plus a query engine — and the medallion (bronze/silver/gold) layering that organizes raw to refined.

• Lake vs warehouse vs lakehouse
• Object storage + table formats
• Medallion layering
• Query engines (Trino/Spark)

Build itDesign a lakehouse for clickstream data: where do raw events land, how do they become clean tables, and what queries the gold layer?

✓ CheckpointExplain what a table format adds on top of object storage, and why a lake without one is hard to trust.

P1 · Advanced · 60 min

Bad data is worse than no data — it erodes trust silently. Learn data quality dimensions (completeness, freshness, uniqueness), assertion tests, anomaly detection, and contracts between producers and consumers. A pipeline that runs green but ships wrong numbers has failed.

• Quality dimensions
• Assertion & schema tests
• Anomaly detection
• Data contracts

Build itOvernight a source silently dropped 30% of rows. Which test catches it before the dashboard does — and how do you alert, not just log?

✓ CheckpointName a data error that passes every schema check and still makes the dashboard wrong.

P2 · Advanced · 55 min

At scale, you must know where data came from, who can see it, and what depends on it. Learn lineage (column-level provenance), cataloging, access control, and PII handling. Governance is what lets a big org trust and safely share its data.

• Column-level lineage
• Data catalog
• Access control & PII
• Compliance (GDPR)

Build itA column must be deleted for GDPR. Use lineage to find every downstream table and dashboard it touches before you drop it.

✓ CheckpointExplain why column-level lineage matters the day someone asks where a PII field ended up.

P3 · Advanced · 55 min

Data platforms need the same observability as services — plus data-specific signals: freshness, volume, and schema drift. Learn pipeline monitoring, SLAs on when tables update, and how to page on a stale mart before the business notices.

• Freshness / volume / schema monitors
• Pipeline metrics
• Data SLAs
• Alerting on staleness

Build itA downstream mart must be ready by 8am. Set an SLA and the freshness monitor that pages if the upstream job runs late.

✓ CheckpointExplain what a freshness SLA promises and who notices first when it is missed — ideally you, not the business.

P4 · Advanced · 65 min

Big tables demand smart physical layout. Learn partitioning, clustering, file sizing (the small-files problem), and how partition pruning turns a full scan into a targeted read. This is where query cost and speed are won or lost at the storage layer.

• Partitioning & clustering
• Partition pruning
• Small-files problem
• Compaction

Build itA daily query scans a year of data to read one day. Partition the table so it prunes to a single day — and fix the tiny-files fallout.

✓ CheckpointExplain how the small-files problem develops and why it makes a well-partitioned table slow.

P5 · Advanced · 50 min

Cloud data platforms bill for compute and scanned bytes, and costs sneak up fast. Learn warehouse sizing, query cost attribution, incremental processing, and caching. The senior skill is delivering the same numbers for a fraction of the spend.

• Compute vs storage billing
• Bytes-scanned optimization
• Incremental processing
• Cost attribution

Build itA dbt job re-processes all history nightly. Make it incremental so it touches only new data — and estimate the cost you just saved.

✓ CheckpointExplain how bytes-scanned billing turns a missing partition filter into a large invoice.

P6 · Advanced · 60 min

Change Data Capture streams every insert/update/delete from a source database into your platform in near real time, without heavy batch pulls. Learn CDC (log-based), streaming upserts, and materializing real-time views. This is how modern platforms stay fresh to the second.

• Change Data Capture (CDC)
• Log-based replication
• Streaming upserts
• Real-time materialized views

Build itMirror a production Postgres table into the warehouse within seconds of each write. Why is log-based CDC better than polling for changes?

✓ CheckpointExplain why log-based CDC beats polling for changes, and what it demands from the source database.