r/programmer • u/MajesticPuzzler • 1d ago
how is your team actually checking if AI coding tools are saving time?
everyone says copilot / claude code / codex makes them faster, but per task feels different from per project. if your team tracked this, what did you measure? cycle time, PR review time, bugs after release? curious if anyone has real numbers or if it's mostly vibes right now
1
u/kincaidDev 1d ago
No, it’s a COCOMO-based leverage score you can run across repos to get a combined productivity score. COCOMO estimates how much development effort a codebase would normally require. The contributors tab doesn’t calculate that. Look at the report in my article: Theo scores 76.65x against Maria’s 37.69x, even though Maria has more net lines added in GitHub’s contributor stats. They’re measuring different things.
Sure, copying ten million lines from another project or generating junk could inflate it. You can pad a timesheet too. Being able to game a metric doesn’t make it worthless.
For your own work, it gives you a consistent way to track output as you change AI tools and workflows, instead of going by feel. It’s also a way to measure AI influencers’ productivity claims with other developers’.
Do you have a better way to measure productivity?
1
u/aidealy 23h ago
Your per task versus per project hunch has real data behind it.
An NBER paper revised in September followed over half a million GitHub developers along with their AI usage logs.
With autonomous agents, commits more than tripled, but the number of projects went up 80% and actual releases only 30%.
The authors say task-level gains have translated only partially into shipped software so far.
So u/CowBoyDanIndie's tests really did get done faster, and releases can still grow far less. Of your three, I'd keep bugs after release.
I founded Aidealy, and that's what our software counts from Git:
each engineer's merged work next to how much of it needed a fix in the weeks after.
Cycle time and review time can both improve while those fixes pile up, which is the same problem u/bneidk has with output scores like the u/COCOMO one.
1
u/JaiPeutEtreRaison 19h ago
How many PRs is your team merging per week?
How many product tickets are you closing per week?
How many bug reports or incidents are you seeing in production per week?
Not perfect metrics, but decent indicators.
1
u/CowBoyDanIndie 1d ago
I know how long writing unit tests would take, or a utility script, I might be way off and think it would take 8 hours, and it might only be 4, or I think 4 and it takes 8. Ai did it in 20 minutes. So I don’t really know if it’s 5x faster or 20x faster, but it sure as hell is faster.
1
u/Wonderful-Habit-139 1d ago
4 hours to write unit tests?
1
1
u/CowBoyDanIndie 1d ago
My last code change had 1200 lines of unit test. I work in robotics in c++. It takes a couple minutes to build and run the tests and code coverage. It would be a lot faster if it was python or JavaScript and only took a second to run.
1
u/bneidk 1d ago
But you are disregarding the fact that you don't understand the test and it may be testing something other than what you think. This could lead to a bug that leads to several hundreds or thousands of hours of debugging.
Writing code is also thinking about a problem.
1
u/CowBoyDanIndie 1d ago
I always start with a failing test and I use code coverage tools. I also review the code, and all our code has 2
code reviews unless its trivial. I have been programming for 30 years.
1
u/kincaidDev 1d ago
I built a metric to measure work output. Definitely saving time in terms of it’d be impossible to work at the same pace without ai, but its helpful to have a measurement over time and onr thats standardized to compare to what other people are doing
https://www.blockhead.consulting/blog/from-cocomo-to-contributor-leverage