A new study by nonprofit AI research group METR suggests that AI coding tools may not deliver the productivity gains many developers expect, particularly for experienced programmers working on complex projects. The randomized controlled trial involved 16 experienced open-source developers completing 246 real tasks on large code repositories. While developers predicted AI tools would reduce completion time by 24%, the study found the opposite: AI increased completion time by 19%.

Researchers recruited developers who had at least 5 years of experience of working on mature code repositories (on average, these were 10 years old and featured about 1,100,000 lines of code). Participants submitted a list of real tasks they had to work on, and they were asked to estimate how long it would take them to complete each task with and without AI assistance. The estimated times are used to determine difficulty levels, and the tasks are randomly categorized as allowing and disallowing AI assistance. For those tasks that allowed AI use, developers were expected to use it if it proved helpful. Crucially, this means they could choose not to use AI if they consider it would not help.

In total, the participants completed 136 tasks with AI assistance, and 110 without. Using the previously reported estimates, the researchers found AI use had increased the time required to complete the tasks by 19%, even though, on average, they expected to see a 24% reduction of the required time before they completed the tasks, and a 20% reduction once the tasks were finished. Interestingly, despite all developers being told they could choose not to use AI or stop using it if it proved unhelpful, participants often stuck to using AI in the tasks where it was allowed. This led the researchers to list "over-optimism about AI usefulness" as one of the factors driving the slowdown in the participants' productivity.

The research team behind the study provides a detailed analysis of the experiment's results, including a disclaimer that they are not drawing any strong conclusions from the study, for instance, that no developer would benefit from using AI in any context, or that AI tools haven't made any significant progress. The researchers claim their findings are consistent with other studies indicating that developers benefit from AI when working on newer or unfamiliar repositories. They also highlight that there are independent reasons to believe AI coding tools will continue to improve, and that there could be contexts where even experienced coders working on mature repositories could benefit from using AI.