The cleanest result in the METR trial is also the one most likely to be misread.
In a randomized controlled trial posted in July 2025, METR reported that experienced open-source developers took 19% longer to complete real coding tasks when they were allowed to use AI tools. After the study, those same developers estimated, on average, that AI had made them 20% faster.
This is one study, not settled consensus. It was run in a particular setting: 16 experienced open-source developers, 246 tasks, mature repositories, and AI tools available between February and June 2025. METR itself now labels the result as historical, saying it no longer necessarily reflects current AI tools or current developer workflows.
That caveat matters. The finding is not that AI cannot help programmers. It is that, in this trial, self-reported speed and measured completion time pointed in opposite directions.
What METR actually tested
METR, short for Model Evaluation and Threat Research, is a nonprofit research organization that evaluates AI systems. The study was released as the paper Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, by Joel Becker, Nate Rush, Elizabeth Barnes and David Rein. METR also published a plain-language write-up of the result.
The study design was unusually close to real work. The developers were not asked to solve toy problems or timed coding puzzles. They worked on issues from open-source repositories they already knew well. On average, they had about five years of prior experience with the projects involved.
Each task was randomly assigned to one of two conditions: AI allowed or AI disallowed. When AI was allowed, developers could use the tools they preferred. In practice, METR says they primarily used Cursor Pro with Claude 3.5 or Claude 3.7 Sonnet, along with chat, agent mode and autocomplete features.
The tasks were real bug fixes, features and refactors. They averaged about two hours each. Developers recorded their screens and reported the implementation time required.
That design makes the result more useful than a broad survey asking whether AI feels helpful. It also keeps the result narrow. The trial studied experienced contributors in mature, high-quality open-source codebases. It does not automatically describe junior programmers, greenfield projects, enterprise teams, one-off scripts, low-stakes prototypes, or later generations of tools.
The measured slowdown
Before starting their tasks, developers expected AI access to reduce completion time by 24%. After the study, they still estimated that AI had reduced completion time by 20%.
The measured result went the other way. METR found that allowing AI increased completion time by 19%.
The sample was small in people and larger in tasks: 16 developers and 246 completed issues. Because each developer worked across both conditions, the researchers could compare AI-allowed and AI-disallowed work within the same pool of participants rather than comparing two entirely separate groups.
METR’s write-up says the slowdown persisted across a range of checks and alternative analyses. The paper also examined possible explanations, including experimental artifacts, project quality standards, developer experience with AI, and the kind of tasks being attempted.
The result should still be handled carefully. A 19% slowdown in this setting is not a universal estimate of AI’s effect on coding. It is an estimate from this trial, using these developers, these tasks, these repositories and early-2025 tools.
Why perception and time could diverge
The gap between the developers’ estimates and the measured result is the part that matters beyond coding.
There are several plausible reasons AI could feel faster while producing slower task completion. A tool can make individual moments feel easier while adding overhead across the whole job. It can draft code quickly, but the developer may then spend time reading, checking, correcting, adapting and integrating that output. It can reduce the unpleasantness of starting a task while quietly increasing review time later.
In experienced codebases, much of the work is not typing. It is understanding implicit constraints, preserving style, noticing edge cases, respecting tests, maintaining abstractions, and making changes that will survive review. A model can be useful and still miss some of that context. The cost of finding and repairing those misses may fall on the human developer.
There is also a psychological problem. If AI produces something quickly, the visible first step looks accelerated. The later checking may feel like normal engineering work, not like AI overhead. A developer can therefore remember the assistance more clearly than the time spent validating it.
That does not make the developers foolish. It makes the study a warning about measurement. Human impressions of productivity are noisy, especially when a tool changes the texture of work rather than simply making the same work shorter.
What the study does not say
METR was explicit about limits that are easy to lose in retelling.
The study does not show that AI tools fail to speed up most developers. It does not show that AI tools fail in other domains. It does not show that future AI systems will fail in the same setting. It does not rule out better workflows, better prompting, better scaffolding, repository-specific fine-tuning, or use cases where AI is more valuable.
It also does not mean developers should ignore AI if they find it useful. Productivity is only one dimension of tool use. A developer might use AI because it makes work more pleasant, helps with exploration, lowers friction, teaches unfamiliar APIs, or makes some tasks less tedious. Those benefits may matter even when a narrow timing measure does not improve.
But the trial does challenge a common assumption: that perceived speedup is a reliable substitute for measured speedup. In this case, it was not. The developers believed the tool had made them faster after completing the study, even though the measured task times showed the opposite.
The result is already aging
AI coding tools changed quickly after the study period. METR’s trial covered tools available from February to June 2025. By early 2026, the organization had started a follow-up experiment and then announced it was changing the study design because the new data had become hard to interpret.
In that February 2026 update, METR said later results showed some evidence of speedup, but selection effects made the central estimate unreliable. More developers were reluctant to participate if they might have to work without AI, and some avoided submitting tasks they especially wanted AI for. That means the later experiment may have missed the people and tasks where AI was expected to help most.
This is an important correction to the simple story. The 2025 trial is strong evidence about one moment and one setting. It is weaker evidence about AI coding tools now, and weaker still about what they will become.
The more durable lesson is methodological. If a tool is important enough to reshape work, it is important enough to measure with something better than vibes. Surveys can tell us what workers feel. Benchmarks can tell us what models can do under simplified conditions. Field trials can show what happens when real people use real tools on real tasks. None is complete on its own.
A productivity claim should survive contact with time
The METR result is uncomfortable because it cuts against a familiar technology story. A tool can be capable, impressive and widely liked while still failing to shorten a specific kind of work in a specific setting.
That is not a contradiction. It is a reminder that software development is not only code generation. The work includes judgment, memory, review, coordination and responsibility for consequences. AI may help some parts of that process while slowing others.
The striking part of the study is not just that experienced developers were slower with AI. It is that they thought they had been faster. For anyone trying to understand AI’s effect on work, that gap may be the more important finding. Productivity claims should not end at the feeling of acceleration. They have to survive contact with the clock.