Inference
Running a trained model forward to produce an output — as opposed to training, which updates its weights.
Inference is a model doing its job: given fixed, already-trained weights, compute a forward pass to produce a prediction or generate tokens. It’s a fundamentally different engineering problem from training — optimized for low latency and high request throughput rather than gradient computation — which is why techniques like quantization, continuous batching, and speculative decoding exist specifically for the inference side of the stack.
Worked example: running a trained model to produce outputs (vs. training it) — for an LLM this is two phases: prefill (process the whole prompt in parallel, compute-bound) then decode (generate tokens one at a time, memory-bandwidth-bound). Gotcha: the two phases have opposite bottlenecks, which is why serving optimizations target them separately — batching helps prefill throughput, while decode is limited by KV-cache memory bandwidth, the reason paged-attention and speculative decoding exist.