Perplexity
How surprised a model is by text — the classic intrinsic measure of language-model quality.
Perplexity is the exponentiated average negative log-likelihood of a sequence: a perplexity of 10 means the model was, on average, as uncertain as choosing among 10 equally likely tokens. Lower is better. It tracks pretraining quality well but correlates loosely with downstream usefulness — which is why evals moved to task benchmarks and human preference.
Worked example: perplexity is the exponential of the average negative log-likelihood — a model that gives the true next token probability 1/10 on average has perplexity 10, as ‘confused’ as choosing uniformly among 10 options. Gotcha: lower is better, but it only measures next-token prediction on the test text, not usefulness, and it is not comparable across different tokenizers.