OpenAI details its defenses against adversarial distillation
OpenAI says new safeguards cut attacks that pull hidden reasoning from GPT-6.2 Astra to under 1% success in its own tests, and that it acts against thousands of accounts a week.
Key takeaways
- OpenAI says new safeguards cut attacks recovering hidden reasoning from an earlier request from 54.0% to 0.2% in its own tests.
- Defenses include adversarial training, classifiers that check tool-call arguments, and tying reasoning to the conversation that produced it.
- OpenAI says weekly enforcement against distillation campaigns has ranged from thousands to tens of thousands of accounts.
- The company repeated allegations involving DeepSeek and Moonshot AI; the post includes no response from either.
- OpenAI says it has told Congress it supports the MATCH Act, the Chip Security Act and the AI OVERWATCH Act.
OpenAI has described how it tries to stop others from training copies of its models on their outputs without permission, a practice it calls adversarial distillation, and says new safeguards in GPT-6.2 Astra and GPT-6.2 Sol sharply cut such attacks in its tests on Astra. The Decoder also reported on the post.
Distillation is a widely used method where a model is trained on another model's outputs. OpenAI argues the unauthorized form is a safety problem as well as an intellectual property one, because a copied model can gain advanced skills, for example in cybersecurity or biology, without the safeguards that limit how the original uses them.
What did OpenAI change?
The company says it puts particular weight on protecting its models' protected reasoning, the internal chain of thought they produce before answering, because those traces can be especially valuable training data. In an internal adaptive test using automated red-teaming, attacks that tried to recover reasoning from an earlier request succeeded 54.0% of the time on GPT-6 Astra under its original safeguards, OpenAI said, compared with 0.2% on GPT-6.2 Astra with the updated ones. For reasoning from the current request, the rate fell from 22.9% to 0.9%. The Decoder reports the same figures. These are OpenAI's own measurements; NextFish has not independently verified them.
The defenses work at two levels, OpenAI said. For single requests, it trains models to refuse extraction attempts, uses input and output classifiers that also check tool-call arguments, and ties a model's reasoning to the conversation that produced it, so rewriting an earlier message drops the reasoning that followed. It also blocks older model families from reading the newer models' protected reasoning. The company says it is now applying some of these improvements to GPT-6 Astra.
At the campaign level, OpenAI says it investigates coordinated extraction attempts. Each week its enforcement has hit anywhere between thousands and tens of thousands of accounts, a pace that rose significantly this past summer, it says, and it shares indicators of compromise with other model providers.
Why does OpenAI say reasoning traces matter?
In an internal study, OpenAI trained a student model on coding and defensive cybersecurity work from Astra-level teacher models. Once fine-tuned, the student that learned from exact reasoning traces generally beat the one that learned from summaries, the company said. Following a round of reinforcement learning, the exact-trace student led only at the largest inference budget tested, and the summary-based student did better at smaller budgets. OpenAI says the results suggest that protecting reasoning traces can limit how much cyber capability transfers, even when the original task is allowed under its policies.
What does OpenAI say about other companies?
OpenAI says it briefed Congress in February on activity indicating DeepSeek was trying to distill U.S. frontier models, and that in September it attributed a core part of a reasoning-extraction campaign to people associated with Moonshot AI, which makes Kimi. These are OpenAI's allegations; the post does not include responses from either company. OpenAI says it backs legislation that would help the U.S. government find and sanction foreign actors that illicitly copy capabilities from U.S. models. It has told Congress it supports three bills: the MATCH Act, the Chip Security Act and the AI OVERWATCH Act.
The Decoder notes that Anthropic has also been fighting distillation attacks and banning tens of thousands of accounts.
Disclosure: NextFish's AI drafting desk uses Claude, a model made by Anthropic.
Questions readers are asking
What is adversarial distillation?
Training a second model on a frontier model's outputs without permission so it gains similar abilities, which can then be deployed without the original's safeguards, as OpenAI describes it.
Which models have the new safeguards?
GPT-6.2 Astra and GPT-6.2 Sol, according to OpenAI, which says some improvements are also being applied to GPT-6 Astra.
Has OpenAI accused anyone of distilling its models?
OpenAI says it told Congress in February about activity indicating DeepSeek was trying to distill U.S. models, and in September it linked part of a campaign to people associated with Moonshot AI. The post includes no response from either company.
Sources
- #openai
- #distillation
- #ai-safety
- #reasoning
- #gpt-6-2
Get the day's top AI & tech stories on WhatsApp: Follow the NextFish channel
Get stories like this every morning.
Related
AI2 min
Thinking Machines releases Inkling 2 Mini, a looped open-weights model
The 26.4B-parameter model runs its middle layers three times per token and needs about 20GB of memory, letting it run on laptops, the company says.
AI2 min
AI-made proof of Generalized Riemann Hypothesis compiles, awaits review
Anthropic's Levent Alpöge says its models produced the proof. Independent checkers say the code compiles, but Scott Aaronson expects understanding it to take years.
AI2 min
Mistral opens Asia-Pacific headquarters in Singapore, plans 200 staff
The French AI company will build custom models for public services, robotics and manufacturing, betting on regional demand for open-weight models it pitches as 'sovereign AI.'
Chips & Infra2 min
OpenAI says its models improved AMD's flagship GPU design by up to 14%
OpenAI says the work delivered about a year's worth of progress in less than five months. AMD confirmed smaller gains on its next CPU, OpenAI says.