Handbooks  /  The SQL Handbook
Handbook~16 min readIntermediate
Deep Dive

SQL, joins,
and the index that saves you.

Anyone can write a SELECT. The difference between "it works" and "it works at scale" is understanding what the database does underneath: how joins combine tables, how an index turns a scan into a lookup, and how transactions keep concurrent chaos correct. Here's the mental model.

01

The relational model

A relational database organizes data into tables — rows (records) and columns (fields) with a fixed schema. Two ideas make it "relational." A primary key uniquely identifies each row (a user's id). A foreign key in one table references the primary key of another, encoding a relationship (an orders row has a user_id pointing at users).

This lets you store each fact once and link records instead of duplicating them: a user's name lives in one place, and a thousand orders reference it by id. Relationships come in shapes — one-to-many (a user has many orders), many-to-many (students and courses, via a join table). The whole power of SQL is querying across these relationships, which is what joins are for.

02

Schema design — normal forms & denormalization

Every schema eventually runs into the same question: when the same fact could appear in five different rows, do you store it once and point at it, or copy it into every row that needs it? The first is a master copy in a filing cabinet — go look it up when you need it. The second is a photocopy left on every desk — faster to grab, but now there are copies to keep straight. Normalization is the discipline of keeping one master copy; denormalization is the deliberate decision to make photocopies anyway. The normal forms are just a precise way of naming which duplicates are dangerous.

Start where almost every schema starts: someone models the business as one wide table, the way a spreadsheet would.

-- orders: one flat table carrying the whole world on every row
order_id  customer_email  customer_name  customer_city  item_skus  product_name  unit_price  category  category_manager
1001      ana@shop.io     Ana Reyes      Lisbon         "77, 91"   Trail Shoe    129.00      Footwear  M. Okafor
1002      ana@shop.io     Ana Reyes      Lisbon         "77"       Trail Shoe    129.00      Footwear  M. Okafor
1003      bo@shop.io      Bo Tran        Porto          "91"       Wool Sock      18.00      Apparel   J. Lindqvist

Nothing here is wrong yet — every query you can imagine is answerable, and none of them need a join. What's wrong is that three unrelated facts (who Ana is, what a Trail Shoe costs, who manages Footwear) are now stored once per order. Duplication isn't a tidiness problem; it's a correctness problem, and it shows up as three specific failure modes.

The anomalies normalization exists to prevent

AnomalyWhat goes wrong in the wide table
Update anomalyAna moves to Porto. Her city is stored on every order she ever placed, so a correct update has to rewrite all of them. Miss one row and the database now holds two contradictory answers to "where does Ana live?" — and no constraint can tell you which is true.
Insert anomalyYou want to record a new product, or that J. Lindqvist now manages a new category. There's nowhere to put it: the only row type is an order, so the fact can't exist until somebody buys something.
Delete anomalyYou delete the last Apparel order. You also just erased the only record that Apparel exists and who manages it. Deleting one fact silently destroyed a different, unrelated one.

These three are why Codd's 1970 paper and the normal forms that followed exist at all. The forms aren't academic hygiene — each one removes a specific class of anomaly.

What 1NF, 2NF and 3NF actually say

1NFOne value per cell. No lists, no repeating groups, no item_skus = "77, 91". Our table fails immediately: you can't index, join, or constrain a comma-separated string. Fix: one row per item, in an order_items table.
2NFEvery non-key column depends on the whole key. Once order_items is keyed on (order_id, product_id), product_name and unit_price depend on the product half only — a partial dependency. Fix: those columns belong in products.
3NFNo non-key column depends on another non-key column. category_manager isn't a fact about the order at all — it's a fact about the category, reached through another non-key column. That's a transitive dependency. Fix: a categories table.

The old mnemonic compresses all three: every non-key column must depend on the key, the whole key, and nothing but the key. Applied to our spreadsheet, that mechanically produces this schema:

CREATE TABLE customers (
  id           bigserial PRIMARY KEY,
  email        text NOT NULL UNIQUE,
  name         text NOT NULL,
  city         text NOT NULL
);

CREATE TABLE categories (
  id           bigserial PRIMARY KEY,
  name         text NOT NULL UNIQUE,
  manager      text NOT NULL
);

CREATE TABLE products (
  id           bigserial PRIMARY KEY,
  sku          text NOT NULL UNIQUE,
  name         text NOT NULL,
  list_price   numeric(10,2) NOT NULL,
  category_id  bigint NOT NULL REFERENCES categories(id)
);

CREATE TABLE orders (
  id           bigserial PRIMARY KEY,
  customer_id  bigint NOT NULL REFERENCES customers(id),
  placed_at    timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE order_items (
  order_id     bigint NOT NULL REFERENCES orders(id),
  product_id   bigint NOT NULL REFERENCES products(id),
  qty          int NOT NULL CHECK (qty > 0),
  PRIMARY KEY (order_id, product_id)
);

All three anomalies are now structurally impossible. Ana's city lives in exactly one row, so it cannot disagree with itself. A product or a category can exist before anyone orders it. Deleting an order can't take a category down with it. You didn't add discipline — you removed the ability to be inconsistent, which is a much stronger guarantee than a code review.

When denormalizing is the right engineering call

The bill arrives on the read path. Rendering one line of a customer's order history now touches five tables:

SELECT o.id, o.placed_at, p.name, p.list_price, c.name AS category
FROM   orders o
JOIN   order_items oi ON oi.order_id   = o.id
JOIN   products    p  ON p.id          = oi.product_id
JOIN   categories  c  ON c.id          = p.category_id
JOIN   customers   cu ON cu.id         = o.customer_id
WHERE  cu.id = $1
ORDER  BY o.placed_at DESC
LIMIT  20;

One join is nearly free. A five-way join on a page served ten thousand times a minute is a different animal: more rows touched, more index descents, and a plan the optimizer can get wrong as statistics drift (you can watch that happen in the query planner lab). Three situations genuinely justify copying data in:

Hot read pathRead far more often than written — a product name changes twice a year and is read a million times a day. Copying it onto the order line removes a join from your busiest query and changes nothing about how often you pay to maintain it.
AnalyticsStar schemas are denormalized on purpose — a wide fact table with pre-joined dimension attributes exists because analytical scans read whole columns and can't afford a join per row. Warehouses optimize for scan throughput, not update correctness.
Point-in-time factsNot really duplication at all — the price an item sold for is a fact about the order, not about the product today. Copying unit_price onto order_items is strictly correct; a join to products would silently rewrite history when prices change.

So the denormalized version of our hot path is a narrow, deliberate edit — two columns, one table, for one query:

ALTER TABLE order_items
  ADD COLUMN product_name text          NOT NULL,  -- copy of products.name
  ADD COLUMN unit_price   numeric(10,2) NOT NULL;  -- price AT PURCHASE TIME

-- the same page, now two tables instead of five
SELECT o.id, o.placed_at, oi.product_name, oi.unit_price
FROM   orders o
JOIN   order_items oi ON oi.order_id = o.id
WHERE  o.customer_id = $1
ORDER  BY o.placed_at DESC
LIMIT  20;

The bill you just signed

Denormalizing doesn't remove the tradeoff, it moves the cost from reads to writes, and you pay it in two currencies. The first is write amplification: renaming one product used to be a single-row UPDATE; now it rewrites every order_items row that copied the name — plus every index entry on those rows, plus the write-ahead log, plus the bytes shipped to every replica. A cheap write became an unbounded one, scaling with how popular the product is. (That amplification is the same force that shapes storage engines — see the B-tree vs LSM race.)

The second is worse: you now own consistency yourself. A foreign key can enforce that a reference exists; nothing in SQL can enforce that a copy still matches. The database will happily hold a row saying "Trail Shoe" forever after the product was renamed. Whatever keeps the copies in step is now your code — a dual write inside the same transaction, a trigger, a materialized view you refresh, or a reconciliation job that sweeps for drift — and every one of those has a failure mode and a staleness window you have to name out loud. This is the same bargain a cache makes; denormalization is just caching baked into the schema, with the same invalidation problem and none of the TTL.

→ The working rule

Normalize until it hurts, then denormalize on purpose — one measured query, one column at a time, with the sync mechanism written down next to it. An integrity bug from a copy that drifted is far harder to find months later than a slow join is to optimize today.

Where you land also depends on the engine. Document stores denormalize by default — that's the whole shape of the SQL vs NoSQL trade — and relational engines differ in what they give you for free (Postgres vs MySQL on materialized views, generated columns, and index-organized tables). Once copies live on more than one node, keeping them in step stops being a schema question and becomes a CP vs AP one.

▶  Watch it explained

Normalization vs denormalization: one master copy, or photocopies everywhere?

03

Joins — combining tables

A join combines rows from two tables by matching a condition, usually a foreign key equal to a primary key. The join type decides what happens to rows that don't match.

JoinReturnsUse when
INNEROnly rows with a match in both tablesYou want records that have related data on both sides
LEFTAll left rows + matches (NULLs where none)Keep all left rows even without a match (users with no orders)
RIGHTAll right rows + matchesSame, mirrored (rarely used; flip the tables and LEFT)
FULLAll rows from both, matched where possibleEverything from both sides, gaps as NULL

The most common bug is expecting a LEFT JOIN but writing an INNER JOIN and silently dropping rows that have no match — a user with zero orders vanishes from the report. Pick the type by asking: "do I want to keep rows that have no partner on the other side?"

→ Watch the fan-out

Joining a one-to-many relationship multiplies rows: one user with 5 orders becomes 5 rows. That's expected — but it means COUNT and SUM after a join can double-count. Aggregate carefully, and know your cardinalities.

04

Indexes — the single biggest lever

Without an index, finding rows where email = ... means a full table scan: read every row and check. On a million rows, that's slow. An index is a separate, sorted data structure — almost always a B-tree — that maps a column's values to the rows containing them, so the database can jump straight to matches in O(log N) instead of O(N).

Why a B-tree? It keeps keys sorted and balanced, so lookups, range scans (BETWEEN, >), and ordered reads (ORDER BY) are all fast. But indexes aren't free: they take space, and every INSERT/UPDATE/DELETE must also update the index — so indexes speed reads and slow writes. Index the columns you filter, join, and sort on; don't index everything.

No index

  • Full scan: read all N rows
  • O(N) per lookup
  • Fine for tiny tables only

B-tree index

  • Sorted structure → jump to matches
  • O(log N) lookup + fast ranges/ORDER BY
  • Costs storage + slower writes

The gap is enormous and exact. On a million rows a full scan checks all million; a B-tree descends its height, ⌈log₂N⌉ ≈ 20 steps:

scan = N = 1,000,000  ·  index = ⌈log₂N⌉ ≈ 20  →  ~47,000× fewer comparisons

But an index isn't always the fast plan: for a query returning most of the table, a sequential scan beats a million random index fetches — which is exactly the tradeoff the planner weighs. The runnable version below counts both and lets the planner choose.

05

The query planner & EXPLAIN

You write what you want (declarative SQL); the database's query planner decides how to get it — which indexes to use, join order, and algorithm. It estimates the cost of each plan using table statistics and picks the cheapest. The same query can run in milliseconds or minutes depending on the plan.

EXPLAIN (and EXPLAIN ANALYZE) shows you the plan the database chose. The single most important thing to spot: a sequential scan on a big table where you expected an index scan — that usually means a missing index, or a query written so the index can't be used (a function on the column, a leading wildcard LIKE '%x', an implicit type cast). Reading EXPLAIN is how you turn "the query is slow" from a mystery into a fix.

→ First move when a query is slow

Run EXPLAIN ANALYZE. If you see a sequential scan over a large table on a filtered column, you're usually one index (or one query rewrite) away from a huge speedup.

06

Transactions & ACID

A transaction groups several statements so they succeed or fail as a unit. Relational databases guarantee ACID:

AtomicityAll or nothing — every statement commits, or the whole transaction rolls back. No half-done transfers.
ConsistencyValid → valid — constraints (keys, checks) hold before and after; the DB never enters an invalid state.
IsolationNo interference — concurrent transactions don't corrupt each other (tuned by isolation level, next).
DurabilitySurvives crashes — once committed, the data is safe even if the server dies.

The classic example is a money transfer: debit one account and credit another must both happen or neither. Atomicity makes that safe; durability makes it stick. This is why financial and inventory systems live on relational databases — ACID is the guarantee.

07

Isolation levels & the anomalies

Perfect isolation (every transaction as if it ran alone) is expensive, so databases offer isolation levels that trade correctness against concurrency. Each level prevents certain anomalies:

AnomalyWhat happens
Dirty readYou read another transaction's uncommitted change (that may roll back)
Non-repeatable readYou read a row twice and get different values (another txn committed a change between)
Phantom readYou re-run a query and new rows appear (another txn inserted matching rows)

The levels — READ UNCOMMITTED, READ COMMITTED, REPEATABLE READ, SERIALIZABLE — progressively prevent more of these. Most databases default to READ COMMITTED (no dirty reads), and you raise it where you need stronger guarantees. SERIALIZABLE prevents all anomalies but costs the most concurrency. Pick the weakest level that keeps your logic correct.

08

The common pitfalls

A handful of mistakes cause most SQL performance pain.

N+1 queriesOne query per item in a loop — fetch a list, then a query per row. Fix with a JOIN or a batched IN (...).
Missing indexFiltering/joining on an unindexed column → full scans. Add the index (check with EXPLAIN).
SELECT *Fetching all columns — wastes I/O and can defeat covering indexes. Select only what you need.
Non-sargableWrapping the column in a function (or leading LIKE '%x') stops the index being used. Rewrite so the raw column is compared.

Underneath all of them is one habit: know what the database is actually doing. Read the plan, index intentionally, fetch in bulk not in loops, and reach for transactions when correctness matters. That's the difference between SQL that works on your laptop and SQL that works in production.

RUN IT YOURSELF

Index vs scan — and why the planner sometimes picks the scan

An index turns an O(N) full scan into an O(log N) B-tree lookup — on a million rows that's 20 comparisons instead of a million, a ~47,000× cut you can watch a real binary search make. But indexes tax every write, and they're not always the fast plan: this lets the query planner choose, and for a query returning most of the table it correctly picks a sequential scan over millions of random index fetches. Change the row count or how many rows match.

CPython · WebAssembly
Frequently asked

Quick answers

What is a database index?

A sorted structure (usually a B-tree) that lets the database find rows by a column's value in O(log N) instead of scanning the whole table. It speeds reads and range/ORDER BY queries, at the cost of storage and slower writes.

INNER vs LEFT JOIN?

INNER returns only rows matched in both tables; LEFT returns all left-table rows plus matches, filling NULLs where the right table has none — so it keeps unmatched left rows that INNER would drop.

What does ACID mean?

Atomicity (all-or-nothing), Consistency (valid state to valid state), Isolation (concurrent transactions don't corrupt each other), Durability (committed changes survive crashes) — the guarantees of a transaction.

What is the N+1 problem?

Running one query for a list of N items and then one query per item — N+1 total. It kills performance and is fixed by fetching related data in a single JOIN or a batched query.

What are 1NF, 2NF and 3NF in plain language?

1NF: one value per cell — no comma-separated lists or repeating groups. 2NF: every non-key column depends on the whole primary key, not just part of a composite key. 3NF: no non-key column depends on another non-key column (no transitive dependencies). The mnemonic covers all three — every column must depend on the key, the whole key, and nothing but the key.

What are update, insert and delete anomalies?

They're the three failure modes duplication creates. An update anomaly is changing a fact in one copy and leaving others stale, so the database holds contradictory answers. An insert anomaly is being unable to record a fact because there's no row to put it in — you can't add a product until someone orders it. A delete anomaly is losing an unrelated fact as a side effect of deleting a row. Normalization removes all three by structure, not by discipline.

When should you denormalize?

When a specific read path is measurably the bottleneck and the copied data is read far more often than it's written — a hot query that would otherwise do an N-way join, an analytics/star schema built for scan throughput, or a point-in-time fact like the price an item actually sold for. The cost is write amplification (one logical update rewrites every copy, its indexes, the WAL and replication traffic) and owning consistency yourself, since a foreign key can enforce that a reference exists but nothing enforces that a copy still matches.

▶  Watch it explained

Database indexes: how CREATE INDEX makes a query 1000× faster

Finished this one? 0 / 208 Handbooks done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

More Handbooks