Model Serving
The infrastructure that runs a trained model behind an API for real-time or batch predictions.
Model serving is the infrastructure that takes a trained model and exposes it for predictions — typically behind an API for real-time (online) inference, or as a batch job over a dataset. It handles loading the model, batching requests, scaling, versioning, and monitoring (TensorFlow Serving, TorchServe, Triton, vLLM for LLMs).
Worked example: a fraud model is deployed to a serving system that loads it on GPU, batches incoming transactions for throughput, autoscales with traffic, and exposes /predict — the app just calls the endpoint. Gotcha: serving has different constraints than training — latency and cost dominate, so dynamic batching, quantization, and (for LLMs) continuous batching and KV-caching matter; and a served model needs drift monitoring, because unlike code it can silently get worse as the world changes.