A model extraction attack systematically queries a target model (often via its public API) and uses the input-output pairs to train a substitute model that approximates its behavior — no access to weights required. It’s a real threat to any team monetizing a proprietary model through an API, since the API surface that makes a model useful is the same surface that makes it queryable at scale.
Worked example: an attacker queries a deployed model many times and uses the input/output pairs to train a cheaper copy that mimics it — stealing the capability without the training cost, or probing to reconstruct training data. Gotcha: rate-limiting and output rounding raise the cost but cannot fully prevent it; a funded attacker with API access can approximate any black-box model, which is why the moat is rarely the weights alone (it is data, distribution, and the product around the model).