How to measure whether AI is speeding up your engineering team
Percent adoption tells you the tools are installed. Four delivery measures, a baseline, and a per-task-type view tell you whether AI is making the team faster or only noisier. A practical setup any engineering leader can run in a month.
Most engineering organizations report AI adoption as a percentage of developers using the tools. It is a reasonable first number and a poor last one. Engineers on public forums have said it plainly: leaders push tool-use mandates without quantifying the impact the tools have on delivery. When the metric is logins, the result is logins.
Four delivery measures, a baseline, and a view by task type replace it. The setup takes an afternoon.
Measure delivery, not usage
Usage metrics (seats, daily active users, suggestions accepted) describe the tool. Delivery metrics describe the team. Four of them cover most of what changes when AI enters the workflow:
- Cycle time. From the moment a ticket starts to the moment its code is merged. This is the headline number. If AI is working, it goes down for the task types where AI helps, and stays flat elsewhere.
- Review turnaround. Time from pull request opened to first meaningful review, and to approval. AI-generated code that nobody trusts shows up here first: pull requests get larger, reviews get slower, and the cycle time gain evaporates in the queue.
- Rework rate. The share of merged changes that get reverted, reopened, or followed by a fix within two weeks. This is the quality check on the speed gain. Faster with higher rework is not faster.
- First-pass acceptance. The share of pull requests approved without a round of requested changes. It captures whether the team's review standard and its AI usage are aligned.
A fifth, optional measure is the split of engineering time by task type: new features, bug fixes, tests, documentation, refactors. AI tends to move the mix before it moves the totals. A team whose test and documentation work quietly drops from a fifth of its time to a tenth has changed, even if cycle time has not moved yet.
Take a baseline before the tools change anything
The most common mistake is measuring after adoption and comparing against memory. Six weeks of history from your ticket tracker and your repository is enough for all four measures, and every later conversation turns from opinion into comparison. Nobody argues with their own before-and-after.
If the rollout already happened, take the baseline now anyway. The trend from here still tells you whether the practice is improving.
Look per task type, not per team
AI does not accelerate everything equally. Boilerplate, tests, migrations and documentation move first. Complex changes in an old, undocumented part of the system move last, and sometimes get slower while the team learns where the tools mislead. A single team-wide number blends both and reads as "no change" when there is a large change hiding inside.
Tag tickets by type, or use the labels you already have, and read the four measures per type. The result usually says exactly where to invest next: which parts of the codebase need documentation and tests so AI can work there, and which task types are ready to be delegated by default.
Watch for the two false positives
Volume without throughput. More pull requests, more commits, more lines. If cycle time and rework do not move, the team is producing motion, not delivery.
Speed with a hidden bill. Cycle time drops, then review turnaround climbs and the senior engineers become a bottleneck because they are the only ones who trust themselves to review AI output. The fix is a shared review standard, which brings the measure back and spreads the load.
Who sets this up
In teams that get it right, one person owns the measurement, the delegation list and the review standard together, because the three move as a set. That is a large part of what a forward deployed engineer does in the first month inside a team: take the baseline, set the per-task-type view, and then change the numbers from inside the codebase, ticket by ticket, with the team watching. The related question of what to do after the AI mandate lands has the same answer: measure delivery, own the practice.
Frequently asked questions
What metrics show whether AI is making developers more productive?
Cycle time from ticket to merge, review turnaround, rework rate within two weeks of merging, and first-pass acceptance of pull requests, read per task type against a baseline taken before the rollout.
Is AI adoption percentage a useful metric?
As a rollout metric, yes: it tells you the tools are installed and people onboarded. As a productivity metric, no: it measures usage, not delivery, and it can reach 100 percent while cycle time and rework stay exactly where they were.
How long before AI shows up in delivery metrics?
Task types that AI handles well (tests, boilerplate, documentation, migrations) typically move within the first weeks of disciplined use. Complex work in undocumented parts of a system moves later, after that code becomes AI-ready with context and tests.
Why did our review times get worse after adopting AI coding tools?
Larger pull requests and code that reviewers do not fully trust. It is the most common side effect of adoption without a shared review standard, and it is fixed by agreeing what gets delegated to AI and how its output is reviewed before it ships.
Part of: Forward Deployed Engineers at Viaro