Sonar
Measuring a lab from outside
What METR could and could not see when it assessed AI-driven R&D at Anthropic.
Draft This topic has been selected for coverage. What you are reading is a first AI research draft based on my direction, published early so you can see what is coming. I have not reviewed it yet, so treat it as provisional. Once the final version is reviewed and published, this notice will be removed.
On 5 November METR published an assessment based on access to Anthropic’s internal data and about 15 staff interviews. Its staff think Anthropic is now improving its models on well-specified tasks at about twice its earlier pace, and that most of the speed-up comes from AI itself. The report also says that Anthropic redacted most of the evidence METR found most informative, so readers have to rely on METR’s judgement.
What METR did
From 12 August to 29 October 2026, METR staff ran what the report calls a high-access evaluation. They asked three questions: has Anthropic’s internal capability growth sped up, how much of that is down to AI, and could bottlenecks slow it soon. They saw internal benchmark data, estimates of algorithmic progress, compute, headcount, code and inference-spend data and internal surveys, and interviewed about 15 staff. They did not get query access to unreleased models.
The work was contracted with Anthropic, with no fees paid in either direction. Anthropic could redact non-public information. METR could say that redactions existed and whether its findings depended on redacted material.
What it found
- Pace: METR’s staff believe that as of October 2026 Anthropic improves its models on well-specified tasks at about twice its March 2024 to October 2025 rate. They would not be surprised by anything from 1.3 to 3 times.
- Benchmark trend: Anthropic’s internal version of the Epoch Capabilities Index rose about 1.9 times faster since October 2025 (95% interval 1.5 to 2.8). Adding frontier models from other labs lowers the estimate to about 1.4.
- Not more compute: METR is confident the speed-up did not come from faster growth in training compute.
- Because of AI: it estimates that roughly 80% of the current speed-up is due to AI, with substantial uncertainty, and that non-AI factors probably drove much of the acceleration in late 2025.
- Bottlenecks: none convincing in algorithms, compute, experiments or data over the next 12 months, although METR’s view shifted several times during the work.
- Autonomy: AI is not yet making highly significant, fully autonomous contributions to the most important training work, but researchers reported early signs that this could come soon.
What it could not show
METR is open about the limits, and they are large:
- Anthropic redacted most of the evidence METR found most informative, so readers cannot easily check it.
- Access was narrower than staff access, and METR reversed its interpretations several times as context arrived.
- A current Anthropic model scores above 50% on all 15 benchmarks used, so the measure may not detect further gains.
- Staff polls put self-reported uplift at about 12 times for the most capable internal model. METR calls that very loose evidence.
- Anthropic removed a compute-allocation verification part of the project midway.
- METR used Claude for much of its data collection and analysis. It found no sign of bias but says fuller access would strengthen the assessment.
- One team member had discussed a role at Anthropic and holds equity in a competitor as a former employee. Another person checked that work. METR says no staff held Anthropic equity at the time, but one incoming staff member does.
- A checklist from the AI Evaluator Forum shows unmet items for this evaluation: no query access, the redaction authority Anthropic held, and no responsible-disclosure policy.
What is established, and what is not
- Established: METR’s staff beliefs and the numbers above, as METR reports them. They come from a trend fit on an internal index, not from an independent benchmark.
- Not established: the 80% attribution, the absence of bottlenecks (METR’s own view moved during the work), and whether the pace carries over beyond well-specified tasks.
- Not checkable from outside: the redacted evidence, and anything that depended on it.
A second look from the same examiner
In August METR reconstructed the OpenAI–Hugging Face incident, also on the lab’s premises and also with redaction rights for the lab. I wrote it up in Containment, not rebellion. The two reports ask different things, one about events and one about a trend, but both show how much an outside examiner depends on what the lab lets it see. METR says so plainly in both.
What this means if you plan around AI progress
This is my reading, not METR’s:
- Read “twice as fast” as a signal about one lab’s research on well-specified tasks. It says nothing about how much faster your own team gets with these tools.
- The most useful parts for outsiders are the method and the candour. The numbers are provisional, and METR says as much.
- When someone reports on a lab, ask what they were allowed to see. Access decides how far a finding can be trusted.