How to Be Wrong

Identify common measurement traps in AI productivity studies.
Explain why self-reported productivity is an unreliable metric.
Recognize selection bias and novelty effects in before/after designs.
Diagnose which methodological mistake a given study is making.

Suppose your manager asks you next week to demonstrate that the AI coding tools your company just purchased are worth the subscription cost. What would you measure? Lines of code generated? Tickets closed? A survey asking whether developers feel more productive? Each of those answers is wrong in a specific way. This lesson describes twelve of the most common mistakes currently being made in published research and in organizations. At least one of them is probably something you, your team, or your employer is doing right now.

Counting Lines of Code Generated

Proxy metrics stand in for concepts that are hard to measure directly; lines of code is one of the oldest
AI assistants generate more code, not necessarily better outcomes. A team that sees a 40% increase in lines of code per developer after adopting AI tools has measured verbosity, not productivity
Deleting 2,000 lines of tangled logic and replacing it with 200 clean ones is an improvement that looks like a loss on this metric [Sadowski2019]
More code means more to read, maintain, and debug, all of which are future costs

Timing Artificial Tasks

A widely cited study found developers using GitHub Copilot completed a task 55% faster than those who did not [Peng2023]
The task was implementing an HTTP server in JavaScript, from scratch, in ninety minutes, by developers with no other obligations that day
Real software development involves navigating a large codebase you did not write, understanding a requirement described ambiguously in a ticket, coordinating with three colleagues, and attending four meetings
Speed on a greenfield toy task does not predict speed on any of that

Measuring Only the Easy Half

AI tools make code generation faster: that half is easy to measure
The other half: time spent reviewing AI-generated code for correctness, time lost debugging confidently wrong suggestions, security vulnerabilities introduced by plausible-looking but insecure code, and technical debt from suggestions that solved the immediate problem while ignoring the surrounding design
More code generated means more code to review: if AI increases code volume without increasing review capacity, cycle time may worsen [Forsgren2021]
Optimizing one stage of a pipeline while ignoring the others is a systems-thinking failure dressed as a productivity study
A study of GitHub Copilot's code found that a substantial fraction of generated code contained security vulnerabilities, and that developers under time pressure accepted insecure suggestions at higher rates [Pearce2022]
Measuring only the inputs that go up while ignoring the costs that also rise is not measurement: it is marketing

Before/After With No Control Group

You roll out AI tools in January, and by June pull requests are shipping faster, so the tools are working, right?
Also between January and June: you hired three more developers, rewrote the CI pipeline, and switched cloud hosting services
Without a group that did not receive the tools, you cannot separate the AI effect from any of the other changes that happened at the same time
Internal validity requires a credible counterfactual, i.e., some way of knowing what would have happened otherwise

Asking Developers If They Feel More Productive

Survey results like "87% of developers report feeling more productive with AI tools" are regularly cited as evidence that the tools work [Liang2024]
Three things make self-report systematically misleading here:
- The Hawthorne effect: people work differently when they know they are being observed and evaluated
- The novelty effect: new tools feel faster because they are novel, but the effect typically fades within weeks
- Social desirability bias: respondents tend to say what they believe the survey wants to hear, especially when management chose the tool
Feeling more productive is not the same as being more productive, and the gap between the two is exactly what measurement is supposed to close
Effects that actually matter emerge over months, not weeks
- Skill atrophy for tasks now delegated to the AI
- Accumulation of technical debt from wrong suggestions
- Changes in how teams collaborate

Goodhart's Law Revisited

In 2023, McKinsey proposed measuring individual developer productivity using counts of commits, pull requests, code reviews, and similar activities [McKinsey2023]
But remember Goodhart's Law: when a measure becomes a target, it ceases to be a good measure [Goodhart1984]
When developers know their commit count is tracked, they make more, smaller commits; when ticket counts are tracked, tickets get split; the numbers improve while the underlying work does not [Beck2023]
Activity is not output, and output is not value

Treating Adoption Rate as a Success Metric

"We have achieved 90% AI tool adoption across engineering" is a procurement outcome, not a productivity outcome
Adoption measures whether the tool is installed and opened
- It says nothing about whether suggestions are useful, whether developers accept them thoughtlessly, or whether the accepted suggestions are correct
Adoption is easier to measure than benefit, which is exactly why it gets reported instead
High adoption combined with low suggestion quality produces a workforce spending time managing a tool rather than benefiting from one

Comparing Volunteers to Non-Volunteers

Studies that compare developers who chose to use AI tools against those who did not are comparing two different populations, not two conditions
Early adopters differ from late adopters and non-adopters in ways that directly predict productivity
- They are more motivated to experiment
- More comfortable with new tooling
- More likely to already be high performers
Selection bias means that any observed difference between the groups may be a property of the person rather than the tool
This is the most common design flaw in industry AI productivity reports

Treating Suggestion Acceptance Rate as a Quality Signal

AI coding assistants commonly report what fraction of their suggestions developers accept
- Higher acceptance rates are presented as evidence that the tool is useful
Acceptance measures whether the generated code looked plausible enough for a developer to press Tab
It does not measure whether the code was correct, secure, or maintainable
Developers under time pressure accept more suggestions, including insecure ones [Pearce2022]
- A tight deadline makes acceptance rate rise for exactly the wrong reasons
- "Adopt AI or lose your job" doesn't encourage careful reflection

Comparing AI to Nothing

Studies that compare AI-assisted developers to a control group using nothing have chosen a baseline that does not exist in practice
The relevant question is whether AI tools outperform the alternatives developers already have
- Developers without AI assistants (still) use Stack Overflow and talk to colleagues
Choosing a weak baseline makes any tool look good

Check Understanding

Your company surveys developers three weeks after rolling out an AI coding assistant. Seventy-eight percent report feeling more productive. Your manager says this proves the tools are working. Identify two specific flaws in this conclusion.

First, the survey was conducted three weeks after adoption, which falls within the novelty period: developers are typically more engaged and positive about new tools during the initial weeks, and that enthusiasm inflates self-reported productivity regardless of actual performance. Second, feeling more productive is not the same as being more productive. Self-report surveys measure perception, which is also distorted by the Hawthorne effect and social desirability bias.

A study compares fifty developers who volunteered to use an AI assistant against fifty who did not. The AI group closes 25% more tickets per week. What is the most likely alternative explanation for this result, and what design change would address it?

The most likely alternative explanation is selection bias: developers who volunteered to use the tool differ from those who did not in ways that predict productivity independently of the tool. A design change that would address this is random assignment: randomly give the tool to half the developers rather than letting them self-select, so that the two groups are comparable on average.

The following measurement plan contains at least two flaws. Identify them and describe a better approach: "To evaluate our AI tools, we will measure lines of code committed per developer per week, comparing the two weeks before and two weeks after tool adoption."

Lines of code has low construct validity as a productivity measure, and a two-week window falls entirely within the novelty period. A better approach would use an outcome-linked metric such as deployment frequency paired with change failure rate, or cycle time from ticket creation to production deployment, and extend the measurement window to at least three to six months. Including a team that did not adopt the tools would provide a control group to separate the AI effect from other concurrent changes.

A vendor announces that their AI assistant's suggestion acceptance rate rose from 25% to 40% over the past quarter and calls this a "quality improvement." What is wrong with this interpretation?

Acceptance rate measures whether a suggestion looked plausible, not whether the accepted code was correct, secure, or maintainable. A rise in acceptance rate can reflect better suggestions, but it can equally reflect increased time pressure, a shift toward simpler tasks where AI does better, or habituation (developers becoming less critical over time). Without a separate measure of the quality of accepted suggestions, such as defect rates or code review rejection rates, acceptance rate tells you nothing about whether the tool is producing better code.

Exercises

Audit Your Own Organization (20 minutes)

List three metrics your organization currently uses (or has used recently) to evaluate the impact of a tool or process change on developer productivity. For each metric, identify which of the failure modes from this chapter it falls into, and briefly explain why. If you believe a metric avoids all of the traps, write one sentence explaining why you think so.

Redesign a Flawed Study (15 minutes)

A company wants to know whether their AI coding tool improves developer productivity. Their current plan is to survey developers about perceived productivity after two weeks, compare responses to a pre-adoption baseline survey, and report the percentage who say they feel more productive.

Identify at least three problems with this design, then describe a better approach in a few sentences, specifying what you would measure, over what time period, and how you would construct a comparison group.

Find the Baseline (10 minutes)

Find a recent vendor claim about an AI tool's impact on developer productivity, such as a blog post, marketing page, or case study. Identify the baseline the claim uses: what does the AI-assisted group get compared to? Then write two sentences describing a more informative baseline and why it would change the interpretation of the result.