How Distillation Attacks Work Against AI Models

TL;DR

• A distillation attack copies a model using only its outputs. The same trick that shrinks a model into a smaller one lets an attacker rebuild its behavior, recover its training data, or strip its safety limits, all through ordinary queries.
• It is cheap. Documented cases run from under $8 to copy a commercial vision API to under $600 for an instruction-following imitator, and less than a dollar to fine-tune a model's guardrails away.
• There is no clean fix, because a model that is useful has to reveal information every time it answers. Defenses raise the cost and help prove theft after the fact, but none of them shut it down.
• For defenders, a model endpoint is one more asset on the graph, with its own reachability, value, and exploit cost. The question is not whether it has a bug, but what an attacker can reach through it and what that costs them.

Knowledge distillation was invented to make models smaller. Point it at a model instead, and the same method copies what that model can do, pulls back the data it was trained on, or peels off its safety limits, using nothing but the answers it was built to give. Here is how distillation attacks work, why they cost so little, and why no single defense shuts them down.

This piece walks through one of the quieter corners of model security: distillation attacks. What makes them hard to dismiss is that they do not rely on a bug. They abuse the one thing a deployed model has to do to earn its keep, which is answer questions.

Knowledge distillation

First described in 2015, it is a way to compress a large, expensive model, or a whole ensemble of them, into a smaller and cheaper one. Rather than training the small model on ground-truth answers alone, you train it to copy the outputs of the big one.

The trick that makes this work is the difference between hard labels and soft labels. A hard label is the ground truth: this image is a dog, written as 1 for dog and 0 for everything else. A soft label is the full set of probabilities the teacher produces, say dog 90 percent, cat 7 percent, cow 2.9 percent, car 0.1 percent. The point is that the soft label carries much more information. A teacher that finds a dog slightly cat-like but almost never car-like is telling you how it generalizes, which is the structure of similarity it learned between classes. This is the "dark knowledge": absent from the hard labels, but holding most of the useful structure.

A soft label's ratios encode how the teacher generalizes, the "dark knowledge" a student can learn from outputs alone. A hard label throws that structure away.

For security, the part that matters is that outputs carry transferable skill. The teacher never hands over its weights or its training data. The student learns from input and output pairs alone. That is precisely the property an attacker turns around. If an honest student can learn a teacher's behavior from its outputs, so can a hostile one. The victim model becomes a teacher against its will.

A few terms before we go on:

• Teacher and student: the model being copied, and the model trained to imitate it.
• Surrogate, knockoff, substitute, imitation model: different names for a student trained to replicate a victim the attacker does not own.
• Logits: raw pre-softmax scores. Soft labels: full probability distributions, a rich signal. Hard labels: just the winning class, a sparse signal.
• Temperature: a dial that controls how soft the distribution is.
• Black-box access: the attacker can only send inputs and read outputs, and never sees the internal weights.

Copying the model: extraction and imitation

In a model extraction attack, sometimes called model stealing, an attacker with nothing but black-box query access (inputs in, predictions out) sets out to duplicate a confidential model. Early work copied models with near-perfect fidelity across logistic regression, neural networks, and decision trees, run against live commercial machine-learning services. It worked even when the API returned only class labels, matching the target on more than 99 percent of the input space at the price of many more queries.

The recipe has two steps:

1. Query a set of inputs against the black-box victim and record its predictions.
2. Train a knockoff model on the resulting input-prediction pairs.

The whole attack in one loop: query the sealed model, keep the input-prediction pairs, train a knockoff, repeat. It still reaches about 99 percent match even when only hard labels come back.

Large-scale extraction has been shown against a range of commercial image and vision APIs, by crafting adversarial examples near the decision boundary and training a substitute through transfer learning. One case reached about 92 percent of the victim's relative accuracy for roughly $7.65 in queries.

Not every copy wants the same thing. Extraction usually aims at one of three goals:

• Task-accuracy extraction: the copy does well on the underlying task, sometimes even fixing the victim's mistakes.
• Fidelity extraction: the copy matches the victim's predictions, errors included.
• Functionally-equivalent extraction: the strongest form, agreeing with the victim on every possible input.

These goals pull against each other. A faithful copy should reproduce the victim's mistakes, while an accurate copy should try to avoid them. Learning-based methods can get close to functional equivalence but never quite reach it. Attacks that recover the weights directly go further still.

For generative LLMs the attack changes, take a strong model's generated text and use it as training data for a weaker, cheaper one. A well-known 2023 demonstration used a leading commercial model to generate 52,000 instruction-following examples, then fine-tuned a small open model of around 7 billion parameters on them. The dataset cost under $500 and the fine-tuning run under $100, so under $600 all in, and the result behaved a lot like its teacher.

But imitation has a ceiling. A cheap copy picks up a teacher's style much faster than its accuracy, so a model that sounds like the original can still be far less reliable underneath. A frontier model copied for a few hundred dollars is not the equal of the one it imitated.

Building a model from scratch takes months, a large team, and serious compute. Copying pieces of one runs from a few hundred dollars down to pocket change.

There is a richer target than the final answer, which is the reasoning that leads to it. Models that show their work emit step-by-step reasoning traces, and those traces are both costly to develop and unusually dense as training material. Copying them, sometimes called chain-of-thought distillation, hands a cheaper model not just what the teacher answered but how it reached the answer. It is the same query-and-train recipe aimed at a more valuable output, which is why teams that ship reasoning models watch for bulk attempts to draw out that step-by-step text.

Reading the training data

Querying a model leaks more than its behavior. It can leak the data the model was trained on. These are privacy attacks, separate from capability copying, but they run on the same black-box access.

Membership inference is the simplest version. Given a record, decide whether it was in the training set. The attacker trains several shadow models that imitate the target, and for those shadows membership is known, then trains a classifier to spot the difference between how a model answers data it saw in training and data it did not. Models tend to be more confident about what they were trained on.

Model inversion goes further and reconstructs representative inputs rather than testing a single one. Given a face-recognition classifier's confidence scores, an attacker can rebuild a recognizable image of someone in the training set by tuning an input to push the model's confidence for that identity as high as possible. The reconstructions are blurry but recognizable, a model leaking the faces it learned from.

Training-data extraction from LLMs is where this gets loud, because large models memorize. By generating many samples and ranking them by how confidently the model reproduces each one, researchers have pulled training text back out. The haul can include names, phone numbers, email addresses, IRC logs, code, and 128-bit UUIDs, even when a sequence showed up in just one training document. Bigger models are more exposed, not less.

Alignment does not close this door. Gigabytes of training data have been extracted from open, semi-open, and closed models. To get past aligned ChatGPT, one method uses a divergence attack: ask the model to repeat a single word such as "poem" or "company" over and over, which knocks it out of chatbot mode and into spilling memorized text. For roughly $200 in queries, this pulled out more than 10,000 unique memorized examples. The trick was disclosed to OpenAI, which patched it, though the broader divergence problem is still open.

The link back to capability theft is clean. Capability distillation copies what a model can do. Data extraction recovers what a model was trained on. Same root fact, that the outputs are a rich and queryable function of the model's internals and training, but a different asset walks out the door.

Removing the safety layer

The branch with the sharpest consequences uses distillation or fine-tuning to strip a model's safety alignment while leaving its capabilities in place.

The idea rests on a structural fact: safety alignment, whether it comes from RLHF (Reinforcement Learning from Human Feedback) or supervised fine-tuning, is a fairly thin layer sitting on top of a capable base model. Distill the capability out of a safety-tuned teacher, or just fine-tune the model a little, and the capability carries over while the refusals fall away.

It takes remarkably little. Fine-tuning an aligned model on as few as ten examples, for well under a dollar, can leave it willing to follow almost any instruction. Even fine-tuning on plainly benign data erodes its safety on its own.

Defenses against model extraction

Defenses fall into four families. None of them is a full answer.

Detection and monitoring tries to catch extraction while it happens by studying query patterns. One approach tracks the distribution of a client's successive queries and raises a flag when it drifts away from normal behavior, since an attacker probing decision boundaries queries very differently from an ordinary user. Detectors like this catch known extraction attacks cleanly, but they have a blind spot: they do not generalize to attacks built on natural-looking data. Newer work runs statistical tests on the embeddings of query traffic to catch extraction against LLM APIs.

Detection is not only a lab exercise. In February 2026, Anthropic reported catching distillation attempts against its Claude models in production at large scale: more than 16 million exchanges routed through roughly 24,000 fraudulent accounts, organized into what it called "hydra cluster" networks that spread traffic across many accounts to stay under per-account limits. The traffic gave itself away through the signature the research predicts: massive volume concentrated in a few areas, highly repetitive structure, and content that lines up with what is most useful for training a model. Anthropic's response leaned on behavioral fingerprinting of API traffic, classifiers tuned to spot bulk attempts to draw out chain-of-thought reasoning, and stronger verification for the kinds of accounts most often abused.

Degradation and output limiting starts from the fact that rich outputs are what make extraction easy, so the provider gives back less: hard labels instead of probabilities, truncated or rounded values, no raw logits. These passive steps help but are not enough on their own. The aggressive, active version, sometimes called prediction poisoning, deliberately perturbs the returned prediction to spoil the gradient signal the attacker's training depends on. It can push the attacker's error rate up by as much as 85 times with little effect on honest users. Related methods return misleading outputs only to queries flagged as out-of-distribution.

Deterrence and raising cost goes after the attacker's economics. One method makes clients complete a computational proof-of-work before they can read each prediction, with the difficulty tuned to how much information the query leaks. That can raise the effort to extract by as much as 100 times while adding only about twice the overhead for regular users. Terms-of-service bans live here too, since most major providers forbid training a competitor on their outputs, though enforcement is hard.

Attribution, meaning watermarking and fingerprinting, gives up on prevention and tries to prove theft after the fact. There are two ways.

• Output watermarking. For LLMs, generation can be nudged toward a pseudorandom "green list" of tokens, which leaves a statistically detectable signature in the text. It can be checked without access to the model's API or parameters and barely touches quality. Text, and any model distilled from that text, can be traced.
• Model watermarking and fingerprinting. A watermark can be embedded straight into a network's weights. Black-box methods followed, using trigger sets, which are special inputs that produce predetermined outputs and can be checked through an API. For LLMs, a secret key-to-response backdoor can be planted that survives downstream fine-tuning, and other methods identify a specific model with a true-positive rate above 95 percent at under 0.2 percent false positives after a single interaction. A wave of recent work tries to tell whether a student was distilled from a particular teacher. The standing caveat: many fingerprints do not survive aggressive continued training, distillation, or paraphrasing.

Every one of these hits the same wall. Output-perturbation and truncation defenses trade robustness against usefulness for real customers. And learning-based extraction can get close to functional equivalence but never quite reach it, which sounds comforting until you remember that for most attackers, close enough is the whole goal.

Why distillation attacks can't be fully fixed

The deepest reason distillation attacks resist a clean fix is structural. A model that is useful has to reveal information through its outputs. Every prediction it returns is, on purpose, a faithful reflection of what it learned. That same reflection is the training signal an attacker uses to build a copy, to infer who was in the training set, to reconstruct data, or to carry capability across without the safety.

You can blur the reflection by perturbing or truncating outputs. You can watch who is looking by monitoring queries. You can charge admission with proof-of-work. You can sign your work with a watermark. What you cannot do is make a model both maximally useful and perfectly opaque. The more your outputs tell users, the more they leak. The more you hide them, the less your service is worth. That tension is the fixed cost of putting a capable model behind an API.

A few caveats

• Attack costs are moving targets. The dollar figures here are illustrative, tied to specific models and moments in time. They show how cheap these attacks can be, not a fixed price, and several of the exploits mentioned have since been patched.
• "Cloning" is overstated. Cheap imitation captures style and surface behavior far more than deep capability or accuracy. A model copied for a few hundred dollars is not the equal of the one it imitated.
• Defenses are contested and adaptive. Every defense here has limits. Query-pattern detectors miss attacks that use natural-looking data, watermarks can be attacked or lost to distillation, and proof-of-work assumes an attacker cannot parallelize cheaply. This stays an active back-and-forth.

Further reading

• Anthropic, "Detecting and preventing distillation attacks" (February 2026), on how one provider detects and slows distillation attempts against a production model.
• MindStudio, "AI Model Distillation Attacks: What They Are and Why They Matter" (March 2026), a plain-language explainer that covers black-box and chain-of-thought distillation.

---