Teach
itself.

A pretrained language model knows a great deal but doesn't reliably do what you ask — it completes text rather than following instructions. Instruction tuning fixes that by fine-tuning on many (instruction, input, output) examples. The catch, circa 2022, was cost: those examples were written by people, task by task, which is slow, expensive, and hard to scale to the breadth of things users actually want.

Self-Instruct (Wang et al., 2022) offered a way around the bottleneck: have the model generate its own instruction data. You seed the process with a small set of human-written tasks — the paper used 175 — and then repeatedly ask the model to invent new instructions in the same spirit, produce inputs and outputs for them, filter aggressively, and fold the survivors back into the pool. Run the loop and 175 seeds bloom into tens of thousands of diverse tasks, enough to fine-tune a base model into a capable instruction-follower for almost no annotation cost. The trick that keeps it from degenerating is a diversity filter.

Self-Instruct is a generate → filter → add loop over a growing pool of tasks:

The loop is self-amplifying in a good way: because each round samples examples from an ever-larger, ever-more-varied pool, the model is nudged toward inventing tasks unlike the ones it's already seen, so coverage expands instead of circling the original 175. For generating the actual answers, the paper also handles classification and free-form tasks slightly differently (generating the output-first or input-first) to keep the instances well-formed. But the engine is simple: the model proposes, a filter disposes, and the survivors become the next round's teachers.

Left unchecked, a model told to "write new instructions" will mostly produce rephrasings of what it just saw — "Write a poem about the sea," "Compose a poem about the ocean," "Create a poem regarding the sea." A dataset full of those teaches almost nothing; the fine-tuned model would be broad in wording but narrow in skill. Diversity, not raw volume, is what makes the synthetic data valuable.

Self-Instruct enforces diversity with a deduplication filter based on ROUGE-L similarity — an overlap score built from the longest common subsequence of tokens between two instructions. For each candidate instruction, it computes the maximum ROUGE-L against every instruction already in the pool, and keeps the candidate only if that maximum is below 0.7. Anything too close to an existing task is discarded before it can dilute the pool. This one threshold is what turns "generate a lot of text" into "generate a lot of genuinely different tasks," and it's the quantitative heart of the method.

ROUGE-L scores overlap by the longest common subsequence (LCS) — the longest sequence of tokens appearing in both instructions in the same order, gaps allowed. From the LCS length you form a precision, a recall, and their F1:

c is the candidate instruction's tokens, r a reference's; |·| is token count. Identical instructions score 1.0; instructions with no common subsequence score 0.0.

The keep/reject rule is then a simple max-and-threshold over the whole pool:

If the candidate is within 0.7 similarity of ANY existing instruction, it's a near-duplicate and dropped. Otherwise it's novel enough to add. The runnable version below implements LCS, ROUGE-L, and this novelty check.

Self-Instruct turned a tiny seed set into a broadly capable instruction-follower, cheaply:

The caveats are real and now well understood. Generated data inherits the teacher's blind spots: whatever the source model is bad at, biased about, or hallucinates, the synthetic set can bake in and amplify. Quality control beyond dedup matters, and the outputs still need care. But as a demonstration that instruction-following can be taught largely from machine-made data, Self-Instruct was decisive.

Self-Instruct lit the fuse on open instruction tuning. Within months, Stanford's Alpaca applied the same recipe — generating ~52,000 instructions in the Self-Instruct style, this time using a stronger teacher model to produce the answers — and fine-tuned LLaMA into a surprisingly capable assistant for a few hundred dollars. A cascade of follow-ups (Vicuna, WizardLM's instruction-evolution, and many others) built on the pattern, and "generate instruction data with a capable model, filter for diversity and quality, fine-tune" became a standard playbook.

More broadly, it's a landmark in synthetic data and model self-improvement. The idea that a strong model can manufacture training data for another connects it directly to knowledge distillation (the teacher's outputs as a training signal) and complements the human-feedback approach of InstructGPT and the AI-feedback approach of Constitutional AI. Today, high-quality synthetic and model-generated data is a central ingredient in training frontier and small models alike — and Self-Instruct is the paper that showed a tiny seed and a filter could go a very long way.