Fact-Check: Do AI Coding Agents Really Make Developers Faster? What METR, DORA and 2025-2026 Studies Actually Show

Fact-Check: Do AI Coding Agents Really Make Developers Faster? What METR, DORA and 2025-2026 Studies Actually Show

Fact-Check: Do AI Coding Agents Really Make Developers Faster? What METR, DORA and 2025-2026 Studies Actually Show

One widely shared number says GitHub Copilot makes developers 55% faster. Another says experienced developers were 19% slower with AI. Both come from randomized controlled trials, and both are real. The debate over AI coding productivity keeps going in circles because people quote headline percentages without the task, population, tooling and measurement behind them. That matters more in late 2026 than it did two years ago. Budgets for agent seats, inference tokens and review tooling now sit next to headcount plans, and executives cite CEO soundbites as if they were measurements. This fact-check takes the five most viral claims, checks each one against the primary sources, and gives a verdict of True, Misleading, False or Mixed. You will also get a map of which study designs measure what, a worked explanation of why lab speedups shrink in production, and a runnable plan for measuring the effect in your own organisation.

What this covers: the Copilot RCT, the METR 2025 trial and its 2026 follow-ups, the Microsoft/Accenture field experiments, Google’s internal RCT, DORA 2024 and 2025, the Stack Overflow 2025 survey, CEO “share of code” claims, GitClear and security studies, SWE-bench versus merge-readiness, and an internal experiment with code.

Context and Background

Three kinds of evidence dominate the AI coding productivity conversation, and they rarely agree because they measure different things. Randomized controlled trials (RCTs) measure the time to complete a defined piece of work with and without AI. Surveys measure what developers believe and report. Telemetry, from repositories and IDEs, measures artefacts such as lines accepted, pull requests merged or code churn. A fourth category, benchmarks such as SWE-bench, measures what a model can do alone against a test suite, with no human in the loop at all.

The incumbents in this debate are well known. GitHub and Microsoft Research published the first widely cited RCT in early 2023. Google’s DORA (DevOps Research and Assessment) program added AI questions to its annual survey in 2024 and published a dedicated “State of AI-assisted Software Development” report in 2025. METR, a non-profit that evaluates frontier AI capabilities, ran the most discussed field RCT of 2025 and then published an unusually candid update in February 2026 explaining why its own follow-up data had become hard to interpret. Stack Overflow’s annual Developer Survey tracks sentiment and trust at scale.

If you already run the delivery and developer-experience metrics described in our guide to DORA metrics and the SPACE framework for platform teams, you have most of the instrumentation needed to test these claims locally. If you are choosing tools rather than measuring them, our comparison of agentic IDEs such as Cursor, Windsurf and Claude Code covers the product side.

The tooling has also changed under the studies. Early RCTs measured inline autocomplete. The 2025 METR trial measured an AI-first editor with chat and agent modes. By 2026 many teams run autonomous agents that open pull requests on their own. Every AI coding agents productivity study cited below measured assistants or agent-mode editors with a human driving, not fully autonomous PR-opening workflows, so treat extrapolation to those workflows with care. A result about one generation of tooling does not transfer automatically to the next, in either direction. That is the first reason to be careful with any single percentage, and it is a recurring theme below. The primary sources are linked throughout, starting with the METR early-2025 study write-up and the Copilot RCT paper on arXiv.

AI Coding Productivity Evidence: What Each Study Design Can and Cannot Tell You

Short answer: the best evidence shows that AI coding tools speed up well-specified, self-contained tasks substantially, deliver smaller and noisier gains on real work in large codebases, and can slow down experts on code they know deeply. Organisation-level delivery gains depend on testing, review capacity and batch size, not on the tool alone.

AI coding productivity evidence map showing six study designs and the metric each one actually measures

Figure 1: Six ways to test a productivity claim, and the metric each design actually produces.

Figure 1 maps a viral claim to six possible sources of evidence. Lab RCTs produce minutes to finish a single greenfield task. Field RCTs produce hours per real issue, including the time to satisfy a project’s own quality bar. Staggered rollouts at employers produce pull requests or completed tasks per developer-week. Surveys produce perceived speed and trust. Telemetry produces artefact statistics such as churn, duplication and suggestion-acceptance rates. Benchmarks produce test-pass rates for a model working alone, which is not the same as mergeable code.

A percentage is meaningless without its denominator

Every productivity percentage is a ratio of two times or two counts. “55.8% faster” in the Copilot RCT is the reduction in average completion time on one task. “26% more completed tasks” in the Microsoft/Accenture experiments is a change in weekly output counts. “19% slower” in the METR trial is the increase in time per issue. These are three different quantities, measured on three different populations, using three generations of tooling. Averaging them, or picking the one that suits a slide, is the root error behind most AI coding productivity claims.

A useful habit is to ask four questions of any number. What was the unit of work? Who did the work? What tool and model were used, and when? And what counted as “done”? The answers usually explain the disagreement without any appeal to bias or hype.

Self-reports and measurements diverge systematically

The most robust finding across the literature is not a speedup or a slowdown. It is that developers’ perception of their own speed differs from measured speed, and not always in the same direction. In the 2023 Copilot RCT, participants estimated an average productivity gain of about 35%, which was lower than the measured 55.8%. In the 2025 METR trial, developers forecast a 24% speedup before starting, and afterwards believed AI had sped them up by 20%, while measured time per issue went up by 19%.

This matters because most organisational evidence is survey-based. The 2025 DORA report announcement says more than 80% of respondents believe AI has increased their productivity. That is a true statement about belief. It is not, by itself, evidence about AI coding productivity as measured in throughput, and DORA’s own analysis is careful to separate the two.

Benchmarks measure capability, not productivity

A model’s SWE-bench score tells you how often it produces a patch that passes the hidden tests for a historical GitHub issue. It does not tell you how often that patch would survive code review, how long a human spent specifying and checking the work, or whether the issue mix resembles yours. Our deeper treatment of trajectory evals, SWE-bench, tau-bench and pass^k reliability covers the evaluation mechanics. In this fact-check, the narrower point is that a benchmark is upstream of productivity. It bounds what an agent can do. It does not measure what a team gains.

The AI Coding Productivity Fact-Check: Five Viral Claims, Five Verdicts

Each claim below gets a verdict, the primary evidence, and a short explanation of why the claim spreads. Verdicts use four labels. True means the claim is accurate as commonly stated. Misleading means a real number is being generalised beyond what it supports. False means the evidence contradicts it. Mixed means credible evidence points both ways.

Claim 1: “AI makes developers 55% faster”

Verdict: Misleading.

This is the most quoted AI coding productivity statistic of all. The number comes from Peng, Kalliamvakou, Cihon and Demirer, “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot” (arXiv, February 2023). The researchers recruited 95 professional programmers through Upwork and randomly assigned 45 to a treatment group with Copilot and 50 to a control group. Everyone had to implement an HTTP server in JavaScript as quickly as possible. Treated participants averaged 71.17 minutes versus 160.89 minutes for the control group, which the paper reports as 55.8% faster. The 95% confidence interval runs from 21% to 89%.

That is a well-run experiment, and the result is real. The problem is what gets dropped when it travels. It was one task, written from scratch, with a clear specification and an automated test suite. It used 2022-era Copilot autocomplete. The completion-rate difference between groups, about 7 percentage points in favour of Copilot, was not statistically significant. The paper also found heterogeneous effects: less experienced developers, those who code more hours a day, and developers aged 25 to 44 benefited more.

Note also the width of the interval. A confidence interval from 21% to 89% means the experiment rules out “no effect” but cannot pin down the size. Quoting 55.8% to one decimal place implies a precision the study never claimed.

Later RCTs with more realistic tasks report smaller effects. Google’s internal enterprise RCT, presented at ICSE 2025’s Software Engineering in Practice track, randomised 96 full-time Google engineers to a logging-service task inside an existing ten-file change. The AI group averaged 96 minutes against 114 for the control group, and the authors’ regression estimate was about 21% faster. They are explicit that the interval is wide and the adjusted estimate is not significant at p < 0.05 (p = 0.086).

So the honest form of Claim 1 is: on a well-specified, self-contained greenfield task, autocomplete-era Copilot roughly halved completion time in one trial. As a general statement about developers, it is misleading.

Claim 2: “AI writes most of the code at big tech companies”

Verdict: Misleading.

The underlying quotes are real, but none of them says “most,” and none comes with a published methodology. On Alphabet’s third-quarter 2024 earnings call on 29 October 2024, Sundar Pichai said that more than a quarter of all new code at Google is generated by AI, then reviewed and accepted by engineers. On the Q1 2025 call in April 2025, he was widely reported as saying the figure had passed 30%. At Meta’s LlamaCon on 29 April 2025, Satya Nadella told Mark Zuckerberg that “maybe 20 to 30 percent of the code that is inside of our repos today” in some projects is written by software. In the same exchange, Zuckerberg predicted that “in the next year probably … half the development is going to be done by AI.” That is a forecast, not a measurement.

Three problems make these figures unsuitable as productivity evidence. First, there is no public definition of “generated by AI.” Does an accepted single-line autocomplete count as AI-written? Does a line the engineer then edits? Nadella also mentioned suggestion acceptance rates of “sort of whatever 30-40 percent,” which suggests the metric is close to accepted-character share rather than a measure of who designed the change. Second, share of code written is not time saved. If AI writes the boilerplate and the human writes the tricky 10%, most of the effort is still human. Third, these are statements made by leaders of companies that sell AI coding products, on investor calls and at product conferences.

For AI coding productivity, the fair reading is that a large and growing fraction of committed characters at Google and Microsoft originates from AI suggestions that engineers accept. That is a meaningful adoption signal. It is not evidence that AI is doing “most” of the engineering, and it says nothing about the productivity effect.

Claim 3: “Experienced developers are slowed down by AI”

Verdict: Mixed.

No AI coding productivity result has been argued over more. This is the METR claim, and it is both better supported and narrower than its critics or fans suggest. In “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” (July 2025), METR recruited 16 experienced open-source developers. They worked on 246 real issues from their own repositories, which average more than 22,000 GitHub stars and more than a million lines of code. Each issue was randomly assigned to “AI allowed” or “AI disallowed.” With AI allowed, developers mostly used Cursor Pro with Claude 3.5 and 3.7 Sonnet, the frontier models at the time. Developers were paid $150 per hour, and the average issue took about two hours.

Issues took 19% longer when AI was allowed. METR’s February 2026 update gives the 95% confidence interval for that estimate as +2% to +39%. The perception gap is the part that went viral: developers expected a 24% speedup and, after finishing, believed they had been 20% faster.

METR is careful about scope. Its authors say the result does not show that AI fails to speed up most developers, or that it will not help in this setting in future. They list caveats: only 16 developers, possible under-experience with Cursor beyond roughly 50 hours of use, rare use of multiple agent trajectories, and a setting of expert developers on large, mature codebases they knew intimately. That setting is the worst case for AI. The developer already holds the context the model lacks, and the project’s quality bar is high.

Then came the follow-up. METR’s update of 24 February 2026 describes a second study starting in August 2025 with 57 developers, ten returning and 47 new, across 143 repositories and more than 800 tasks. For the ten returning developers, the estimate was an 18% speedup, with a confidence interval from 38% faster to 9% slower. For new developers, it was a 4% speedup, from 15% faster to 9% slower. Neither interval excludes zero.

METR calls this data an unreliable signal, and explains why. Recruitment got harder because more developers refused to work without AI. Surveys found that 30% to 50% of developers skipped submitting some tasks because they did not want to do them without AI. Both effects systematically remove the cases with the highest expected uplift. METR says the raw estimate is likely a lower bound and “likely a bad proxy” for real impact. Its developers’ conversations suggest a larger speedup in early 2026 than early 2025, which METR describes as “very weak evidence” of an increase.

The verdict is therefore Mixed. The 2025 slowdown is a real, statistically significant result for a specific population and tool generation. The best follow-up data points toward a speedup, but with wide intervals and acknowledged selection bias. What has held up across both rounds is the perception gap: developers’ own estimates are not a reliable instrument.

Claim 4: “AI-generated code has more bugs and security issues”

Verdict: Mixed, leaning True for security on unreviewed output.

The security evidence is consistent over several years and tool generations. Pearce and colleagues’ “Asleep at the Keyboard?” study (IEEE Symposium on Security and Privacy, 2022) prompted Copilot with 89 scenarios built around high-risk CWEs (Common Weakness Enumerations) from MITRE’s Top 25 list. About 40% of the 1,689 generated programs were vulnerable. Perry, Srivastava, Kumar and Boneh’s user study, “Do Users Write More Insecure Code with AI Assistants?” (ACM CCS 2023), found that participants with access to an AI assistant based on codex-davinci-002 wrote significantly less secure code, and were more likely to believe their code was secure.

Industry testing in 2025 points the same way. Veracode’s 2025 GenAI Code Security Report, published 30 July 2025, tested more than 100 large language models on 80 curated code-completion tasks in Java, Python, C# and JavaScript. It reports that 45% of test cases introduced an OWASP Top 10 vulnerability. Java failed more than 70% of the time, cross-site scripting defences (CWE-80) failed 86% of the time, and log injection (CWE-117) 88%. Veracode’s most uncomfortable finding is that security performance did not improve as syntax accuracy did, and larger models did not do significantly better than smaller ones. Treat the vendor framing with care, since Veracode sells application-security testing, but the direction matches the academic work.

Maintainability is murkier. GitClear’s 2025 report, based on 211 million changed lines from its customers and open-source projects, found that code blocks with five or more duplicated lines rose about eightfold during 2024. It also found that copy-pasted lines exceeded moved lines for the first time, while moved lines fell about 40%. GitClear interprets moved code as a proxy for refactoring. Those trends coincide with AI adoption, but the analysis is observational and cannot isolate AI as the cause. GitClear is also a vendor of code-analysis tools.

Bugs in production are the weakest link in the chain. There is no large public RCT showing higher defect rates for AI-assisted teams once normal review and testing apply. DORA’s survey-based estimate, discussed below, links higher AI adoption with lower delivery stability, which is consistent with more defects reaching production but does not prove it. The fair verdict: unreviewed AI output carries well-documented security risk, and quality risk is plausible but not yet causally established at team level.

Claim 5: “SWE-bench scores equal real productivity”

Verdict: False.

SWE-bench Verified is a 500-task subset of real GitHub issues from 12 Python repositories, human-validated so that each task is solvable and its tests are fair. Top models score above 70% on it, as the SWE-Bench Pro authors note. Two pieces of 2025-2026 evidence show why that number overstates real-world usefulness.

First, contamination and difficulty. Scale AI’s SWE-Bench Pro (arXiv, September 2025) assembled 1,865 tasks from 41 actively maintained repositories, using copyleft-licensed and private commercial code to reduce the chance the solutions sit in training data. At publication, the best model scored 23.3% Pass@1 on the public set, and the best on the commercial set was 17.8%.

Second, mergeability. In a note dated 10 March 2026, METR had four active maintainers of scikit-learn, Sphinx and pytest review 296 AI-generated patches that passed SWE-bench Verified’s tests, blinded and mixed with 47 human-written “golden” patches. Maintainers would have merged only about 68% of the original human patches, which the study uses as the noise baseline. Even after normalising for that, maintainer acceptance ran about 24 percentage points below the automated grader. Roughly half of test-passing AI patches would not be merged. Rejection reasons were code quality and repo conventions, breaking other code, and failing to actually fix the issue. For Claude Sonnet 4.5, METR estimates the task “time horizon” at about 50 minutes by the automated grader but about 8 minutes by maintainer judgement.

METR does not claim a capability ceiling. Agents got one attempt with no reviewer feedback, which is not how humans work. But the verdict is clear: a benchmark score is an upper bound on unattended capability under test-only grading. Equating it with developer productivity is false.

Deeper Analysis: Why AI Coding Productivity Gains Shrink Before They Reach Delivery Metrics

The gap between “55% faster on a task” and “delivery throughput barely moved” is not a mystery. It follows from where coding sits in the delivery pipeline, and from what extra code volume does downstream.

Delivery pipeline showing where AI coding productivity gains appear and where review, CI and stability absorb them

Figure 2: AI accelerates the coding stage most, while extra volume and larger batches load review, CI and production stability.

Figure 2 traces a ticket from pickup to delivered value. AI helps most at “write code.” The dotted edges show the side effects: more volume arriving at review, larger batches arriving at CI, and more change failures arriving at deployment. If review and testing capacity stay fixed, they become the bottleneck, and the coding speedup turns into queue time.

An Amdahl’s-law view of task speedups

Amdahl’s law, borrowed from parallel computing, says the overall speedup is limited by the fraction of work you actually accelerate. Here is an illustrative calculation, not a measured result. Suppose writing code is 30% of the elapsed time for a typical ticket. The rest is understanding context, testing, waiting for review, addressing comments and deploying. Suppose AI makes the coding portion 50% faster, roughly the Copilot RCT effect.

The new elapsed time is 0.70 + 0.30 × 0.50 = 0.85 of the original. That is a 15% reduction in cycle time, from a 50% speedup on the accelerated part. If AI also adds 10% to review time, because reviewers now read more code they did not write, and review is 20% of elapsed time, you add 0.02 back. The net reduction drops to 13%.

Now apply the METR finding about where time goes for experts. If prompting, waiting on the model and reviewing its output cost more than the typing they replace, the “accelerated” fraction can have a speedup below 1. In that case the whole ticket gets slower. The same arithmetic produces both the optimistic and pessimistic headline numbers. What changes is the share of work that is pure code production, and the overhead of checking AI output.

What DORA’s survey models say about the organisation level

DORA is the only large program that connects AI adoption to delivery outcomes across many organisations, using statistical models on survey data. Its findings changed between 2024 and 2025, which is itself informative.

The 2024 DORA report estimated that a 25% increase in AI adoption was associated with a 7.5% increase in documentation quality, 3.4% in code quality and 3.1% in code review speed. It also estimated that higher adoption was accompanied by a 1.5% decrease in delivery throughput and a 7.2% reduction in delivery stability. More than 75% of respondents relied on AI for at least one daily task, and 39% reported little or no trust in AI-generated code.

The 2025 DORA AI report, based on nearly 5,000 survey responses and more than 100 hours of qualitative interviews, reports that 90% of respondents use AI at work, with a median of about two hours a day. Unlike in 2024, AI adoption now shows a positive relationship with delivery throughput and product performance. It still shows a negative relationship with delivery stability. DORA summarises this as AI being an amplifier: “AI doesn’t fix a team; it amplifies what’s already there.”

DORA also published an AI Capabilities Model listing seven practices that magnify AI’s benefits. They are a clear and communicated AI stance, healthy data ecosystems, AI-accessible internal data, strong version control practices, working in small batches, a user-centric focus, and quality internal platforms. Several of these are classic DORA capabilities. Small batches and version control are exactly the controls that stop extra code volume from turning into instability, as Figure 2 suggests.

A caution applies to all DORA numbers. They come from cross-sectional survey data and structural models. They show associations, not causal effects of turning AI on. They are still the best organisation-level evidence available, and they line up with the pipeline argument above.

What the Stack Overflow survey adds

The Stack Overflow 2025 Developer Survey AI section shows the same tension at the individual level. About 84% of respondents use or plan to use AI tools, and about 51% of professional developers use them daily. Yet only about 33% trust the accuracy of AI output, while about 46% distrust it, and just 3% highly trust it. The top frustration, cited by 66%, is “AI solutions that are almost right, but not quite.” About 45% say debugging AI-generated code is more time-consuming. Positive sentiment toward AI tools was about 60%.

“Almost right, but not quite” is the mechanism that connects the surveys to the METR result. Code that is 90% correct is quick to generate and slow to finish. The final 10% needs exactly the deep context an expert has, so the expert pays the cost of reading and fixing generated code without gaining much from the generation step.

Results summary table

The table below lists the primary AI coding productivity results cited in this article. All numbers come from the linked sources. Where a source gave no interval, the cell says so.

Study Design Population Tooling Headline effect Uncertainty
Peng et al. 2023 Lab RCT, one task 95 Upwork developers Copilot, 2022 55.8% faster 95% CI 21% to 89%
Cui et al. 2024-25 Three field experiments 4,867 developers at Microsoft, Accenture, a Fortune 100 firm Copilot +26.08% completed tasks SE 10.3%
Google, ICSE SEIP 2025 Enterprise RCT, one task 96 Google engineers Internal code completion and NL-to-code About 21% faster Not significant at 0.05 after controls
METR, July 2025 Field RCT, real issues 16 expert OSS developers Cursor, Claude 3.5/3.7 Sonnet 19% slower 95% CI +2% to +39%
METR, Feb 2026 update Field RCT, real tasks 57 developers Late-2025 tools 18% faster for returning, 4% faster for new Intervals cross zero, selection bias
DORA 2024 Survey and modelling Global survey respondents Mixed Throughput -1.5%, stability -7.2% with higher adoption Associational
DORA 2025 Survey and modelling Nearly 5,000 respondents Mixed Throughput positive, stability still negative Associational

Methodology caveat: these rows cannot be averaged. They differ in unit of work, population, tool generation and definition of “done.” The Cui et al. row combines three experiments with different rollout designs and adoption rates, and its paper reports that less experienced developers adopted more and gained more.

The field experiment most people skip

The Microsoft/Accenture work deserves more attention than it gets. Cui, Demirer, Jaffe, Musolff, Peng and Salz, “The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers,” covers 4,867 developers at Microsoft, Accenture and an anonymous Fortune 100 company. Access to Copilot was randomised as part of real rollouts. Developers with access completed 26.08% more tasks, with a standard error of 10.3%. Less experienced developers both adopted the tool more and gained more.

This is the strongest causal evidence for a real-world AI coding productivity gain on ordinary work. It also has limits worth knowing. Outcome measures are counts, such as pull requests and builds, rather than value delivered. The tool generation is pre-agent Copilot. And the large standard error means the true effect could plausibly be well under 26%.

How to Run Your Own AI Coding Productivity Experiment

Published AI coding productivity studies tell you the plausible range. Only an internal experiment tells you where your team sits in it. The design below borrows from METR’s per-task randomisation and DORA’s outcome metrics, and avoids the selection problems METR described.

METR style randomized trial sequence for measuring AI coding productivity per issue

Figure 3: The per-issue randomisation loop used in METR’s field trial, adaptable for an internal study.

Figure 3 shows the loop. A developer submits a real issue and a forecast of how long it will take. A randomiser assigns the issue to AI-allowed or AI-disallowed. The developer works to the project’s normal quality bar while time is recorded. The analyst compares measured time across arms, and separately records how much faster the developer believed they were.

Choose the randomisation unit deliberately

Per-task randomisation gives the most statistical power for a small team, because every developer contributes to both arms. It has the selection problem METR found in 2026: developers avoid submitting tasks they do not want to do without AI. Mitigate it by randomising after the task is committed to the sprint, not before, and by tracking which tasks were withdrawn.

Per-developer or per-team randomisation, often a staggered rollout, is closer to how the Microsoft/Accenture experiments worked. It avoids task cherry-picking and captures team-level effects such as review load. It needs many more people to detect a modest effect. If you have fewer than about 40 developers, a staggered rollout with a clear baseline is often more practical than a pure RCT.

Pre-register the metric and the decision

Write down, before the experiment starts, the primary metric, the minimum effect you care about and the decision each outcome will trigger. Good primary metrics are cycle time per task for the speed question and change failure rate or rework rate for the quality question. Secondary metrics should cover the SPACE dimensions (Satisfaction, Performance, Activity, Communication and Efficiency) through a short survey, so you can see the perception gap directly.

Do not use lines of code, suggestion acceptance rate or “percentage of code written by AI” as a primary outcome. They are activity metrics that rise with adoption whether or not value improves.

Analyse log time ratios, not raw minutes

Task durations are right-skewed. A few long tasks dominate any mean. METR-style analyses therefore model the logarithm of time, which turns a multiplicative speedup into an additive coefficient. The sketch below fits a simple model with the developer’s own forecast as a covariate and uses a cluster bootstrap by developer for the confidence interval. It is a starting point, not a substitute for a statistician on a large study.

# Requires: pandas, numpy, statsmodels
# Input CSV columns: dev_id, task_id, ai_allowed (0/1),
#                    forecast_min, actual_min, perceived_speedup (optional)
import numpy as np
import pandas as pd
import statsmodels.formula.api as smf

df = pd.read_csv("ai_trial_tasks.csv")
df = df[(df.actual_min > 0) & (df.forecast_min > 0)].copy()
df["log_actual"] = np.log(df.actual_min)
df["log_forecast"] = np.log(df.forecast_min)

# Developer fixed effects absorb skill and repo differences.
formula = "log_actual ~ ai_allowed + log_forecast + C(dev_id)"
fit = smf.ols(formula, data=df).fit(cov_type="cluster",
                                    cov_kwds={"groups": df.dev_id})
beta = fit.params["ai_allowed"]
print(f"Time ratio AI/no-AI: {np.exp(beta):.3f}")
print(f"Cluster-robust 95% CI: "
      f"{np.exp(fit.conf_int().loc['ai_allowed'][0]):.3f} to "
      f"{np.exp(fit.conf_int().loc['ai_allowed'][1]):.3f}")

# Cluster bootstrap by developer as a robustness check.
rng = np.random.default_rng(42)
devs = df.dev_id.unique()
boot = []
for _ in range(2000):
    sample = rng.choice(devs, size=len(devs), replace=True)
    parts = [df[df.dev_id == d].assign(dev_id=f"{d}_{i}")
             for i, d in enumerate(sample)]
    b = pd.concat(parts)
    if b.ai_allowed.nunique() < 2:
        continue
    r = smf.ols(formula, data=b).fit()
    boot.append(np.exp(r.params["ai_allowed"]))
lo, hi = np.percentile(boot, [2.5, 97.5])
print(f"Bootstrap 95% CI for time ratio: {lo:.3f} to {hi:.3f}")

# Perception gap: what developers believe versus what was measured.
if "perceived_speedup" in df:
    print("Mean perceived speedup:", df.perceived_speedup.mean())

A time ratio of 0.85 means AI-allowed tasks took 15% less time. A ratio of 1.19 matches the METR 2025 slowdown. If the confidence interval spans 1.0, you have not shown an effect in either direction, and you should say so rather than reporting the point estimate alone.

Run a power check before you start. As an illustrative rule of thumb, if per-task log-time has a standard deviation around 1.0 after adjusting for forecasts, detecting a 20% effect with reasonable power needs a few hundred tasks. METR’s 246 issues were enough to reject zero in 2025, but only just. Small teams should plan for a longer study or accept that they can detect only large effects.

From result to decision

Internal AI coding productivity experiment workflow from hypothesis through DORA and SPACE metrics to a decision matrix

Figure 4: An internal experiment pipeline that links speed, quality and experience metrics to a pre-agreed decision.

Figure 4 shows the full workflow. It starts from a real decision, such as buying more agent seats, expanding agent autonomy or restricting tools in a regulated codebase. It runs a baseline of four to eight weeks on DORA and SPACE metrics, then a treatment period with training. It measures speed, quality and experience separately and only then feeds a decision matrix.

Speed result Quality result Experience result Recommended action
Faster, CI excludes zero Rework and change failures flat Positive Scale the rollout, keep monitoring stability
Faster Change failures up Positive Invest in tests, review capacity and small batches before scaling
No clear effect Flat Positive Keep for satisfaction and onboarding, do not budget on speed
No clear effect Worse Mixed Restrict to low-risk task types, add security scanning on AI output
Slower Any Developers believe faster Treat as perception gap, coach workflows, re-test on other task types

The last row is the METR lesson. If measured time goes up while developers report speedups, the survey cannot be your decision input.

Trade-offs, Gotchas, and What Goes Wrong

The first gotcha is task-type mixing. AI coding productivity varies sharply by task. Greenfield scaffolding, test writing, migrations between well-documented APIs and unfamiliar-language work tend to benefit. Deep changes to a mature codebase you know intimately tend to benefit least. An experiment that mixes these without recording task type will produce an average that describes no real task. Tag every task, and report effects per category if you have enough data.

The second is selection. METR’s 2026 update is a case study in how adoption breaks measurement. Once developers depend on a tool, they avoid AI-disallowed conditions, decline to participate, or withdraw the tasks where AI helps most. All three bias the estimate toward zero. The more enthusiastic your team, the more your internal RCT will understate the effect, unless you randomise after task commitment and track withdrawals.

The third is agentic time accounting. With autonomous agents, a developer can start a task, switch to something else while the agent runs, and return to review. Wall-clock time per task can rise even as throughput per developer improves, because work runs in parallel. METR flagged exactly this difficulty in its 2026 data. Measure both elapsed time per task and completed tasks per developer-week, or you will misread parallelism as slowness.

The fourth is the review tax. Every study that measures only the author’s time ignores the reviewer’s. If agents raise PR volume and size, review becomes the constraint, and reviewer fatigue is when security defects slip through. The Perry et al. finding that AI users were more confident in less secure code makes this worse. Track review latency and PR size alongside author speed.

The fifth is metric gaming, the fastest way to corrupt an AI coding productivity programme. “Percentage of code written by AI” is easy to inflate and correlates with nothing you care about. Once leadership targets it, teams will accept more suggestions. Duplication and churn will rise, which is the pattern GitClear reports at industry scale. Keep acceptance metrics as diagnostics, never as goals.

The last is generalising across tool generations. The 2023 Copilot RCT, the 2025 METR trial and the 2026 SWE-bench maintainer study used different tools. A finding about autocomplete says little about agents, and a finding about Claude 3.7 Sonnet says little about the models shipping in late 2026. Re-run your measurement when the tooling changes materially.

Practical Recommendations

Treat published AI coding productivity numbers as a prior, not a forecast. The credible range for well-specified individual tasks runs from roughly 20% to 55% faster in controlled trials. For real work in mature codebases, it runs from modest slowdowns to modest speedups, with wide uncertainty. Organisation-level throughput gains appear only where testing, review and batch-size discipline absorb the extra volume. That is what DORA’s amplifier finding says, and it matches the pipeline arithmetic.

Spend first on the capabilities that convert local speed into delivery: fast automated tests, strong version control practice, small batches, and enough review capacity. Add security scanning that targets the known weak spots of generated code, such as injection, cross-site scripting and log injection. Do this before expanding agent autonomy.

Measure locally with a design you can defend. Pre-register the metric and decision. Randomise per task or stagger rollout per team. Analyse log time ratios with clustered intervals, and capture perceived speed separately so you can see the gap. Re-run when tools change. When you report results upward, lead with the interval, not the point estimate.

Finally, stop quoting share-of-code-written as productivity. It is an adoption metric.

A short checklist:

  • Tag every task with a type and complexity before randomising.
  • Use cycle time and change failure rate as primary metrics, not lines or acceptance rate.
  • Randomise after sprint commitment and log withdrawn tasks.
  • Track reviewer time, PR size and review latency, not just author time.
  • Scan AI-heavy changes for OWASP Top 10 classes before merge.
  • Report confidence intervals and the perception gap to leadership.
  • Re-test after each major model or tool change.

Frequently Asked Questions

Does AI make developers faster?

AI coding productivity depends on the task. On well-defined, self-contained tasks, yes. Controlled trials report time reductions from about 21% (Google’s enterprise RCT) to 55.8% (the GitHub Copilot RCT). Field experiments at Microsoft, Accenture and a Fortune 100 company found about 26% more completed tasks. On real work in large codebases that experts know well, the 2025 METR trial found a 19% slowdown, and METR’s 2026 follow-up points toward modest speedups with wide uncertainty. The honest answer depends on the task, the developer’s familiarity with the code, and the tool generation.

What did the METR developer productivity study find?

METR’s July 2025 randomized trial had 16 experienced open-source developers work on 246 real issues in their own large repositories. Each issue was randomly assigned to allow or forbid AI tools, mainly Cursor with Claude 3.5 and 3.7 Sonnet. Issues took 19% longer with AI allowed, with a 95% confidence interval of +2% to +39%. Developers had expected a 24% speedup and afterwards believed they were 20% faster. A February 2026 update with 57 developers suggested speedups, but METR called that data unreliable because of selection effects.

Is the 55% faster Copilot statistic accurate?

It is accurately reported but widely over-generalised. The 2023 study by Peng and colleagues randomised 95 Upwork developers to build an HTTP server in JavaScript. The Copilot group finished in an average 71 minutes versus 161 for the control group, 55.8% faster, with a 95% confidence interval of 21% to 89%. It was one greenfield task with a clear spec and tests, using 2022-era autocomplete. It does not show that developers are 55% faster on typical work.

What does the DORA AI report say about productivity?

The 2025 DORA report on AI-assisted software development surveyed nearly 5,000 professionals. It found 90% use AI at work and over 80% believe it raises their productivity. Its models show AI adoption now associated with higher delivery throughput, but still with lower delivery stability. DORA calls AI an amplifier of existing strengths and weaknesses. The 2024 report had estimated small drops in throughput (1.5%) and stability (7.2%) with higher adoption. Both are survey-based associations, not causal estimates.

Is AI-generated code less secure?

Evidence consistently shows elevated security risk in unreviewed AI output. Pearce and colleagues found about 40% of 1,689 Copilot-generated programs vulnerable in high-risk CWE scenarios. A Stanford user study found AI-assisted participants wrote less secure code yet felt more confident in it. Veracode’s 2025 testing of over 100 models reported OWASP Top 10 flaws in 45% of test cases, with no improvement in newer or larger models. Review, static analysis and security-focused prompting reduce, but do not remove, this risk.

Do high SWE-bench scores mean an agent will boost my team’s productivity?

Not directly. SWE-bench measures model capability, not AI coding productivity. It checks whether a model’s patch passes hidden tests, without human feedback. METR’s March 2026 study had maintainers review test-passing patches and found roughly half would not be merged, due to code quality, breakage or incomplete fixes. Scale AI’s harder SWE-Bench Pro showed top models scoring under 25% at launch. Use benchmarks to shortlist models, then measure cycle time, rework and review load on your own codebase.

Further Reading

References

  1. Peng, S., Kalliamvakou, E., Cihon, P., Demirer, M. “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot.” arXiv:2302.06590, February 2023. https://arxiv.org/abs/2302.06590
  2. METR. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.” 10 July 2025. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
  3. METR. “We are Changing our Developer Productivity Experiment Design” (uplift update). 24 February 2026. https://metr.org/blog/2026-02-24-uplift-update/
  4. METR. “Many SWE-bench-Passing PRs Would Not Be Merged into Main.” 10 March 2026. https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/
  5. Cui, Z. K., Demirer, M., Jaffe, S., Musolff, L., Peng, S., Salz, T. “The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers.” SSRN 4945566; Management Science. https://papers.ssrn.com/abstract=4945566
  6. Paradis, E. et al. “How much does AI impact development speed? An enterprise-based randomized controlled trial.” arXiv:2410.12944; ICSE 2025 SEIP. https://arxiv.org/abs/2410.12944
  7. Google Cloud. “Announcing the 2024 DORA report.” https://cloud.google.com/blog/products/devops-sre/announcing-the-2024-dora-report
  8. Google Cloud. “Announcing the 2025 DORA report” (State of AI-assisted Software Development). https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report
  9. Google. “2025 DORA AI Capabilities Model report.” https://cloud.google.com/resources/content/2025-dora-ai-capabilities-model-report
  10. Stack Overflow. “2025 Developer Survey: AI.” https://survey.stackoverflow.co/2025/ai
  11. Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., Karri, R. “Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions.” IEEE S&P 2022. arXiv:2108.09293. https://arxiv.org/abs/2108.09293
  12. Perry, N., Srivastava, M., Kumar, D., Boneh, D. “Do Users Write More Insecure Code with AI Assistants?” ACM CCS 2023. arXiv:2211.03622. https://arxiv.org/abs/2211.03622
  13. Veracode. “AI-Generated Code Poses Major Security Risks in Nearly Half of All Development Tasks” (2025 GenAI Code Security Report). 30 July 2025. https://www.veracode.com/press-release/ai-generated-code-poses-major-security-risks-in-nearly-half-of-all-development-tasks-veracode-research-reveals/
  14. GitClear. “AI Copilot Code Quality: 2025 Research,” as reported by DevClass, 20 February 2025. https://www.devclass.com/ai-ml/2025/02/20/ai-is-eroding-code-quality-states-new-in-depth-report/1626250
  15. Deng, X., Da, J. et al. (Scale AI). “SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?” arXiv:2509.16941, September 2025. https://arxiv.org/abs/2509.16941
  16. Fortune. “Google CEO says more than a quarter of the company’s new code is created by AI.” 30 October 2024. https://fortune.com/2024/10/30/googles-code-ai-sundar-pichai/
  17. TechCrunch. “Microsoft CEO says up to 30% of the company’s code was written by AI.” 29 April 2025. https://techcrunch.com/2025/04/29/microsoft-ceo-says-up-to-30-of-the-companys-code-was-written-by-ai/
  18. The Register. Report on Nadella and Zuckerberg remarks at LlamaCon. 30 April 2025. https://www.theregister.com/2025/04/30/microsoft_meta_autocoding/

By Riju — about

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply

Your email address will not be published. Required fields are marked *