METR says public evidence can't rule out drastic AI R&D acceleration soon
The AI evaluation group says models usually match skilled humans on its coding tasks, time horizons double about every 3.6 months, and capabilities are harder to measure.
Key takeaways
- METR says the public evidence cannot exclude a drastic speed-up of AI research within the next couple of years.
- Models usually equal or beat skilled humans on METR's coding and optimization tasks, though it ran formal human baselines on only three tasks.
- Time horizons on METR's combined task suites are doubling roughly every 3.6 months.
- AI agents appear to show worse judgment than human experts on tasks without clear empirical feedback, METR says.
- METR stresses significant uncertainty about which capabilities matter and how fast they are improving.
METR, an organization that evaluates frontier AI systems, says current public evidence is not enough to rule out a drastic acceleration of AI research and development in the next couple of years. By that it means progress so fast that people could no longer meaningfully understand, assess and respond to catastrophic risks from new systems, according to its capability update published on November 20. The Decoder also reported on the findings.
METR says it remains significantly uncertain about central questions, including which capabilities would actually be needed for drastic acceleration and how fast the relevant ones are improving. Its evaluations cover public frontier models through Opus 5.5 and GPT-6 Astra.
What did METR find?
On its coding and optimization tasks, models usually equal or beat skilled humans, METR says. When it recruited its own human baseliners, it typically could not find people who outperform current frontier models, and in a couple of cases with many domain specialists, the models ranked very high but not at the very top. METR cautions that it ran formal human baselines on only three tasks and that its baseliners were probably less skilled in their domains than typical frontier AI researchers.
On its combined task suites, the length of task, measured by how long it takes humans, that models complete half the time is doubling roughly every 3.6 months, METR says; it calls this measure the time horizon. METR publishes a range of uncertainty around that doubling time. It says a naive extrapolation of that trend would have models reaching a time horizon equal to the longest score threshold in its task set, 200 hours, in July 2027. It notes the doubling time is similar to a trend OpenAI recently reported on its internal research tasks.
On a composite index of model ability across the public frontier, METR says it does not see strong acceleration or slowdown, though it sees uncertain signs of trend changes for individual developers and is unsure those findings are valid.
Where do models still fall short?
On tasks that require choices without clear empirical feedback, AI agents appear to show worse judgment than human experts, METR says. In one task, model judges overrated the quality of work produced by a model as strong as or stronger than themselves, compared with an expert scorer. METR adds that on some ML tasks the models did show sound judgment, beating its human baseliners, though it is unclear how far that generalizes.
Models also perform somewhat worse on tasks METR estimates are costlier to turn into training environments, even at the same task length, though they appear to be improving at similar rates, the group says. METR says it is increasingly concerned that its easy-to-run, containerized tests could overstate real-world capability, because similar environments have become common in model training.
Why does METR say measurement is getting harder?
When METR introduced its time-horizon measure in March 2025, the best models could handle tasks of about one hour, the group says. Within a year, models had saturated a suite with tasks of up to 30 hours, and it is now hard and expensive to find human baseliners who beat the models in many areas. METR says public frontier models are now capable enough in areas like coding and hacking that it can no longer argue risk is very low on the basis of what they cannot do on verifiable software tasks alone.
Disclosure: NextFish's AI drafting desk uses Claude, a model made by Anthropic.
Questions readers are asking
What is METR?
An AI research organization that evaluates frontier AI systems, including capabilities relevant to accelerating AI research and development.
What does 'drastic AI R&D acceleration' mean?
METR uses it for progress so fast that people could no longer meaningfully understand, assess and respond to catastrophic risks from new AI systems.
Is METR predicting that acceleration will happen?
No. It says current public evidence cannot rule it out, and that it remains significantly uncertain about the key questions.
Sources
- #metr
- #ai-safety
- #ai-evaluation
- #time-horizon
- #frontier-models
Get the day's top AI & tech stories on WhatsApp: Follow the NextFish channel
Get stories like this every morning.
Related
AI2 min
Lean and Mathlib communities certify AI-assisted Riemann Hypothesis proof
A joint statement says Levent Alpöge's 17-million-line Lean proof is correct after independent checkers confirmed it. Mathematicians still have to understand the ideas behind it.
AI2 min
Etsy says it will rewrite AI training terms as long-time sellers quit
New terms let Etsy and its partners use seller content to train AI. After long-time sellers left, Etsy said it would not train models to copy their work and promised new wording.
AI2 min
Google opens its Genie 4 world model to developers in the Gemini API
Developers can now generate playable 720p worlds with sound from a prompt or image, at about 10 cents per second of output, with sessions capped at two minutes.
AI2 min
OpenAI details its defenses against adversarial distillation
OpenAI says new safeguards cut attacks that pull hidden reasoning from GPT-6.2 Astra to under 1% success in its own tests, and that it acts against thousands of accounts a week.