What kind of metric would you trust for measuring organization gains?

I think number of features released to customers (not behind a feature flag or still being rolled out, but fully rolled out). And number of bug fixes (only those reported by customers).

Also just in general, customer satisfaction, acquisition, conversion, retention, etc.

Number of completed org-level roadmap items, org-level goals achievement rate, and so on.

I also think a good one would be seeing an increase in meeting estimation, like if project was estimated to take X days with Y devs, does the use of AI increased how often you met or beat those estimates in actual time/dev effort?

And you'd want to compare that against prior years, where no AI was used, within the same org, or try going 1 quarter without AI and another with and compare quarter to quarter.

▲

wilkystyle 4 hours ago | parent [-]

I also think something along these lines is the correct answer. It can be hard to pin down an exact metric because once you start optimizing for a metric it tends to not be a good measure of the original thing anymore. But in general I think it comes down to some measure of feature velocity combined with a counter metric on support/maintenance burden.

"Number of PRs merged" seems like "number of lines of code" wearing a trenchcoat, and I thought we all agreed back in the 90s that number of lines of code was a terrible measure of software productivity...

	▲	strange_quark 3 hours ago \| parent [-]
		Feature velocity is another that's extremely easy to game. My company is trying this right now: instead of measuring PRs or lines of code, we are measuring number of customer facing features shipped. Well guess what? Everything is now a customer-facing feature. You did a big internal code refactor and data migration? Well guess what, that's a customer feature now because it unlocks future such and such. Deploy a new piece of infra? Customer feature. Dev tool improvement? Customer feature. IMO trying to measure productivity gains is a fools' errand. The only thing that matters is CSAT, escaped defects, retention, cost per contact, and other metrics that measure actual business outcomes.

▲

jdlshore 4 hours ago | parent | prev | next [-]

Incredibly hard problem, but METR had a good method. They had people estimate how long a task would take (before knowing whether AI would be used), and then randomly assigned each task to “with AI” or “without AI.” When the data was in, they compared actual/estimate ratios of the two populations.

(Presumably, they used a t-test that only compared people against themselves.)

Interestingly, for that study (released in 2025), participants self-rated themselves as 20% more productive, but were measured as being 19% less productive.

▲

jimsperry 3 hours ago | parent | prev | next [-]

Microsoft itself has a system for measuring this which outlines a few example metrics: https://queue.acm.org/detail.cfm?id=3819080

There was a nice talk about this by one of the author's at this year's BUILD conference: https://build.microsoft.com/en-US/sessions/BRK210

	▲	didibus 2 hours ago \| parent [-]
		Great read, thanks for sharing!

▲

dnackoul 4 hours ago | parent | prev | next [-]

We use a tool called Weave (I believe YC 25?) that analyzes PRs for "expert units of work" and shows lift from AI tools. My understanding is they have their own proprietary model that assesses the difficulty of each PR. I find the organization level view and pivots useful and aligned with intuitive expectations.

	▲	Traubenfuchs an hour ago \| parent [-]
		This sounds like voodoo.

▲

jayd16 5 hours ago | parent | prev | next [-]

It'll probably take a really good product built by a profitable company evangelizing an AI workflow with reproducible examples dating back a few years.

▲

cj 5 hours ago | parent | prev [-]

Engineering headcount.