The gauge broke: devs felt 20% faster with AI, measured 19% slower (2025)

jiggawatts

"19 August 2025" This may as well have been written in the stone ages, when we were banging AI rocks together. I just did a ~6 month project in ~2 weeks using a frontier model. I wouldn't even have attempted this kind work a year ago, with or without the AIs available at the time!

tombot

“16 developers across 246 tasks”

bitwize

This study was shown to be flawed at the time; METR has retracted it. And it doesn't take into account current frontier models. AI makes you more productive. This is no longer up for debate. The energy you spend arguing last year's talking points is better spent knuckling down and learning the tools.

shaky-carrousel

There was a study that people using the keyboard instead of the mouse felt they were working faster but in fact they were working slower. A perception thing. Users were more engaged when using a keyboard.

loveparade

These studies are meaningless because speedup is heavily dependent on the kind of work you're doing. No doubt that you can do mechanical refactors 100x faster with AI, and also no doubt that using AI will be slower for tasks where it's less about writing code and more about context/world knowledge or building understanding. Averaging across these tasks doesn't make sense because everyone's work consists of a different distribution of tasks. A frontend dev doing tailwind integration for his day job is gonna see very different speedups than someone working in a niche scientific codebase. Taking the average makes about as much sense as taking the average of the speedup from calculators for a mathematician, a farmer, and an elementary school student.

ianbutler

2025 is such old news that this just isn't relevant. METR already redid the study at a later date and now finds a likely 18% speedup "For the subset of the original developers who participated in the later study, we now estimate a speedup of -18% with a confidence interval between -38% and +9%" (note their use of - and + here could be slightly confusing but they do mean 18% faster per the post) https://metr.org/blog/2026-02-24-uplift-update/

wewewedxfgdf

My two bosses are anti-AI. Whenever I tell them about how awesome AI is, they come back with stories about how they used AI and it couldn't even do anything basic and what it did do had errors. People will always create a world narrative that matches what they already believe. Anti AI people are always quoting these "facts" about how AI reduces productivity even when developers feel it increases productivity - it reinforces their world view.

onion2k

Generation got cheap. Verification got expensive. That proves AI is capable of doing one part of the software engineering process. The 16 devs in the study trusted AI to write the code. Once we trust AI to do the verification as well we'll realise the gains we feel we're getting now. Essentially we're intentionally going slower on the second half because the trust is missing. Alternatively, rather than trusting AI to do the validation, we could follow the vibe-coder approach by skipping the validation entirely, and trust that the generation stage is good enough not to need it. Historically that's come with some small downsides, like the code being a broken mess of security holes, but with time AI might fix that.

make_it_sure

why is this on the front page. It's an old debunked study lol, not relevant at all today. Read more than just the title

jdkoeck

> The honest counter, and it matters here more than usual. This is most likely the dip in a J-curve, not the destination. Oh, the irony of this post being AI-generated.

luckilydiscrete

https://www.faros.ai/blog/ai-software-engineering The actual study with the data, minus the "I was right all along" commentary

vachina

It will feel slower because I finish my task in 1/20th the time and spend the remaining time browsing HN.

raincole

Yeah, when people who are not familiar with AI and use Cursor with Sonnet 3.7 they are only 19% slower. In retrospect that research was very bullish for AI.

orphereus

As I read the blog post, I thought that it was released today. Maybe point out that it is almost a year old. It feels like it is manipulating HN users. And this is coming from an AI sceptic.

DonHopkins

Luc Barthelet, who I worked with at EA, is a Mathematica whiz (he later worked at Wolframe Research on Wolfram|Alpha), and he would prototype game ideas in Mathematica, which would render out web pages with animations. He came up with a fun idea for a racing game renderer: it distorted the perspective transformation a bit, grading depth on a curve, so far away things would linger in the distance a bit longer, then speed up and WHOOSH past you, seeming even faster than they would be photorealisticly! https://www.mobygames.com/person/29352/luc-barthelet/ https://community.wolfram.com/web/luc

miika

Well.. feelings have never been a good way to measure quantities.

bezier-curve

One thing I see missing in a lot of these discussions is whether or not the metric is solely based on speed. I think AI just allows you to look at your code in different ways and provides more chances to catch mistakes. I am definitely slower with AI assistance, but that is because I use it to increase the quality of my work.

sigmoid10

If you linked to the actual source of the study [1] instead of a random blog only talks about the result, you would see the big banner that the authors put there noting that the study is horribly outdated. Current models do make developers faster. [1] https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...

arisAlexis

So why are all the top labs using the tools internally? They are lying or they are stupid? Devs wish that was true but it isn't and it will get better.

xg15

> AI speeds up typing, which was never the bottleneck for an expert in a codebase they already know. For me as a dev, that's not the whole truth. Where I've found actual value in AI (and I think were some of that "perceived speedup" is coming from) is looking up things . Unless you know the codebase and used libraries extremely well, you will have to do lots of "micro-lookups" during coding, where you have to find the specific APIs or library functions for your problem, then figure out how exactly you have to call them, how to handle the result, etc. That's lots of "research" work interleaved with actually writing the code. AIs seem to be good enough to have a lot of that knowledge already baked into their weights, at least for popular platforms, so if you prompt it something, you can skip all that low-level lookup work or at least defer it until code review. Even during review, it's easier, because you don't have to come up with the appropriate library function from scratch, you only have to verify that the ones the AI used make sense and are used correctly.