Self-Supervised Learning
Learning from unlabeled data by creating the labels from the data itself — predict a hidden part from the rest.
Self-supervised learning learns from unlabeled data by inventing a prediction task from the data itself — mask a word and predict it, predict the next token, or predict one view of an image from another. The “labels” come free from the data, so it scales to huge unlabeled corpora.
Worked example: a language model is pretrained by hiding the next token and predicting it over billions of sentences — no human labeling — learning grammar, facts, and reasoning as a side effect; then a little supervised fine-tuning adapts it to tasks. Gotcha: self-supervision is why foundation models exist (labels no longer gate scale), but the choice of pretext task shapes what is learned — predict-the-next-token teaches different things than predict-in-feature-space (JEPA) — and it still needs some labeled data or RLHF to align to what you actually want.