Garbage in,
garbage out.

It's tempting to obsess over the model or the retriever and treat data as a given. But in practice the data pipeline is where most AI quality is decided. A great embedding model retrieving over badly-chunked, duplicate-ridden, low-quality passages returns bad results; a modest one over a clean, well-structured corpus returns good ones. The data is the product, and the pipeline that produces it — ingest, chunk, deduplicate, quality-filter — is the real engineering.

This is true for both uses of data. For RAG, the pipeline decides what a retriever can even find: the shape of the chunks, whether duplicates crowd out diversity, whether junk pollutes the results. For training, it decides what the model learns: duplicates get over-weighted, noise teaches bad patterns, and quality filtering is what separates a useful dataset from a large one. Same stages, same discipline. And the humbling truth the pipeline reveals is that most of the raw pile shouldn't survive.

Retrieval and context windows work best on focused passages, not whole documents — so the first transform is chunking: split each long document into smaller windows. A retriever can then return the specific passage that answers a query instead of a giant document where the answer is buried. Chunks usually carry a bit of overlap so a fact that straddles a boundary isn't lost between two chunks.

Chunk size is a genuine trade-off. Too large and each chunk mixes several topics, so a query matches on the wrong part and retrieval precision drops. Too small and a chunk lacks the surrounding context to be meaningful on its own. The right size depends on the content (dense technical text vs. narrative) and the task, and it directly sets how many chunks a document becomes — a sliding window of a given size and overlap turns one document into a predictable number of pieces. Chunking is the stage that multiplies your item count up; the next stages cut it back down.

Raw corpora are full of copies — the same article syndicated across sites, boilerplate headers and footers, near-identical templates. Deduplication keeps one representative of each distinct piece and drops the rest, and it matters more than it looks.

In retrieval, duplicates crowd out diversity: ask for the top 5 passages and get 5 copies of the same thing instead of 5 different relevant ones — a real loss of information. In training, duplicated text is over-weighted, driving memorization, wasting compute on redundant examples, and risking eval-data leaking into training. Dedup — exact-match or near-duplicate detection — restores diversity and ensures every surviving item carries new information. It's one of the highest-leverage, least-glamorous steps in the whole pipeline.

Chunking maps one document of length L to a number of windows of size s with overlap o — roughly the length divided by the step, rounded up:

A 1000-token doc at size 200, overlap 50 (step 150) → about 7 chunks. Smaller chunks or more overlap → more pieces.

Then dedup and quality-filter each keep only a fraction, and — crucially — those fractions compound. The usable dataset is the raw count times every stage's keep-rate:

Multiplicative: two stages that each keep 60% and 50% leave 30%. That shrinkage is the point — you're keeping the good part, not all of it. The runnable version below chunks a doc, dedups, quality-filters, and computes the compounding yield.

The filter stage is where the pipeline's philosophy lives. A few common quality signals:

The recurring lesson across the field — from the Phi models' textbook-quality data to every serious RAG deployment — is that a smaller, cleaner dataset beats a larger, noisier one. Because bad data doesn't just fail to help; it actively hurts, polluting retrieval and teaching wrong patterns. So the engineering goal is not to maximize how much data you keep but how much good data you keep — which means investing in the filters, not just the crawl. The compounding yield isn't a bug to minimize; it's the sign your filters are doing their job.

The first mistake is bad chunk boundaries: splitting mid-sentence or mid-idea (naive fixed-size splitting) produces chunks that are individually meaningless, so retrieval returns fragments. Prefer semantic or structure-aware splitting (paragraphs, sections) and use overlap so nothing important falls in the gap. The second is skipping dedup because "more data is better" — it isn't; duplicates crowd retrieval and over-weight training, and the fix is cheap relative to the harm.

Two more. Over-filtering the other way: an aggressive quality filter can strip legitimate but unusual content and narrow your corpus, so validate that the filter keeps what matters, not just what's average. And no data lineage: when a bad answer traces to a bad chunk, you need to know where that chunk came from — track provenance through the pipeline so you can fix the source, not just patch the symptom. The whole discipline reduces to a mindset shift: treat the data pipeline as the primary engineering surface, measure the yield and quality at each stage, and remember that the goal is the best data, not the most. Get the pipeline right and the model, whichever you pick, has a fighting chance.