Assessing AI

<div class="center">
 <h1>Assessing AI</h1>
 how to tell if AI coding tools are working without fooling yourself
 <img src="../../files/talks/codebender-logo.svg" alt="Third Bit logo" width="20%" />
 <h2>Greg Wilson</h2>
 2026-11-06
 <img src="../../files/talks/cc-by.svg" width="20%" alt="CC-BY" />
</div>

---

## What This Talk Is About

-   In 2023, GitHub announced that its AI tools made developers 55% faster [Peng2023]

-   That number appeared in every executive presentation for a year

-   How would you know whether to believe it?

-   This talk is about:

-   How we get the answer wrong

-   How we can get it right

---

## Why Should You Listen to Me?

-   Forty years writing software and teaching programming

-   Used to run a project trying to connect software engineering researchers and practitioners

-   I am not going to tell you AI-assisted coding doesn't work

-   I am going to tell you how to find out whether it works *for you*

-   Singular and plural

---

## The Real Question

-   "Is AI helping my team?" sounds simple

-   It hides three questions:

-   Helping with *what*?

-   Measured *how*?

-   Compared to *what*?

-   These are not pedantic questions

-   The answer you get depends entirely on how you operationalize them

---

## This Is Hard

-   [Sadowski2019] is an entire book devoted to "this is difficult"

-   Software output is not homogeneous

-   A ten-line bugfix may be worth more than a thousand-line feature

-   Much of software work is invisible

-   Developers spend only about a quarter of their day writing code [Meyer2017]

---

## For Example

-   Prechelt had 73 professional developers solve the same task [Prechelt2000]

-   Completion times ranged from 0.6 to 63 hours (a 105X spread)

-   Controlling for programming language: 17X

-   With a careful definition of "more productive": 5X

-   The answer depends on exactly what question you ask

---

## How We Get It Wrong

-   Four recurring mistakes sit underneath the hype

-   Measuring the easy thing

-   Measuring too small

-   Comparing to nothing

-   Asking people how they feel

-   Each is a way of mistaking *outputs* and *outcomes*

---

## Wrong #1: Measure the Easy Thing

-   A *proxy metric* stands in for something harder to measure directly

-   Lines of code: a 40% increase per developer measures verbosity, not value [Sadowski2019]

-   Commit and PR counts: developers make more, smaller commits when commits are tracked [McKinsey2023]

-   Adoption rate: measures whether the tool is installed, not whether it helps

-   Acceptance rate: measures whether code looked plausible enough to press Tab [Bakal2025]

-   *Goodhart's Law*: when a measure becomes a target, it ceases to be a good measure [Goodhart1984]

-   Activity is not output, and output is not value

---

## Wrong #2: Measure Too Small

-   Timing a 90-minute greenfield task tells you about a 90-minute greenfield task

-   Real work: unfamiliar codebases, ambiguous tickets, coordinating with colleagues

-   Individual speed is not system throughput

-   If review is the bottleneck, more code makes cycle time worse [Forsgren2021]

-   Code generation speed gets measured because it's the easy part

-   The hard part doesn't: review time, debugging confidently-wrong suggestions, security debt [Pearce2022]

-   Five major LLMs evaluated in 2025: none produced web code meeting industry security standards [Dora2025]

---

## Wrong #3: Compare to Nothing

-   Before/after with no control group cannot separate the AI from
    the twelve new engineers, the refactored CI, and the new cloud provider

-   Comparing AI to "nothing" uses a baseline that does not exist in practice

-   Developers without LLMs use documentation, colleagues, and their own thinking

-   The question is whether AI beats the alternatives developers already have

---

## Wrong #4: Ask People

-   "87% of developers report feeling more productive with AI tools" [Liang2024]

-   *Novelty effect*: new tools feel faster because they are new — and that fades

-   *Hawthorne effect*: people work differently when they know they are being watched

-   *Social desirability bias*: respondents say what they think management wants to hear

-   Volunteers vs. non-volunteers:
    *early adopters* are more motivated and more likely to be high performers

-   A two-year study found Copilot users were more active than non-users
        even before the tool was introduced [Stray2026]

---

## Who Does This?

-   *Empirical Software Engineering* studies how software is actually built

-   The field has existed since the 1960s

-   Draws on psychology, sociology, economics, and statistics as well as computer science

-   Bad news: most studies don't address questions practitioners care about [Begel2014]

-   Good news: "most" isn't "all"

-   Bad news: you don't have ESE researchers in house

-   Good news: you (probably) have someone just as good

---

## Already In House

-   You don't have to invent an AI assessment program from scratch

-   UX researchers already have the right intellectual tools

-   They do rigorous *qualitative* research: interviews, think-aloud
        sessions, and thematic analysis [Braun2006, Goodman2012]

-   They do rigorous *quantitative* research: usability metrics and
        the statistics to interpret them [Tullis2013, Sauro2016]

-   They are already trained to ask "does this actually help people do their work?"

-   So turn to your UX research team to create the assessment program

-   The rest of this talk is a tour of their toolkit

---

## Where to Start

-   *Goal-Question-Metric* provides a structured path from intent to measurement [Basili1994]

-   Define the goal: what object, what property, from whose viewpoint, in what context

-   Generate the questions whose answers would tell you whether the goal was achieved

-   Then pick the metric that answers each question

-   Does the metric actually measure what you care about?

-   If possible, get someone *outside* your organization to challenge you on this

---

## A Checklist for Any Study

-   *Conclusion validity*: was the sample large enough? are effect sizes reported?

-   *Internal validity*: was there a control group? random assignment? novelty and learning ruled out?

-   *Construct validity*: does the measurement actually capture the claim?

-   *External validity*: are the subjects representative of the population you care about?

---

## Getting Started

-   Use *qualitative methods* when you do not yet know what to measure

-   They answer "what is happening?" and "why?" rather than "how much?"

-   E.g., a study of 410 developers revealed where AI helps and where it gets in the way [Liang2024]

-   Invisible to any study that only measures task completion times

---

## Interviews

-   Do interviews before creating surveys

-   *Semi-structured interviews* have a guide but allow follow-up

-   Consistent enough to compare, flexible enough to surprise

-   *Open questions* invite narrative; *closed questions* invite classification

-   Avoid leading questions: "Don't you find it faster?" assumes the answer

-   Pilot with 3–5 people before distributing it

-   And make the changes

---

## Thematic Analysis

-   *Open coding*: read through the data and tag segments with descriptive labels

-   *Gerund coding*: "avoiding AI for security tasks" rather than "AI distrust"

-   This preserves what participants are actually doing

-   A theme is a claim you could write as a sentence, not a bucket for related quotes

-   Stop collecting data when new interviews stop introducing new codes [Braun2019]

---

## Controlled Experiments

-   Manipulate one variable and measure its effect while holding others constant

-   Randomization assigns participants to conditions randomly, distributing unknown confounders

-   This is the mechanism that makes causal claims defensible

-   Full *blinding* is rarely possible in software engineering

-   The developers know if they're using AI

---

## Pitfalls

-   *HARKing*: hypothesizing after the results are known

-   I.e., writing the paper as if the pattern you found after analysis was predicted in advance

- *p-hacking*: running twenty analyses and reporting only the one with p < 0.05

-   With 20 independent tests, you expect one false positive by chance

-   *Pre-registration* commits hypotheses before data collection, making both problems visible

-   If authors will not share data, there is a decent chance of errors in the work [Wicherts2011]

---

## Observational Studies

-   Measure the world as it is: real behavior, large datasets, no ethical problem withholding an intervention

-   Disadvantage: you can't establish causation because confounders cannot be ruled out

-   GitHub data is not a representative sample of software development

-   Survivorship bias and "looking where the light is"

-   Projects with more tests also tend to have more experienced developers

-   So you cannot attribute lower defect rates to testing alone

---

## Do It Cheaply First

-   A small, informal study produces better evidence than a meeting [Hall2019]

-   Five sessions are enough to find most problems in a specific workflow [Nielsen1993]

-   Enough for "we observed developers getting stuck here"

-   Not enough for "X% of developers have this problem" or "AI causes more bugs"

-   *Formative evaluation* improves a thing; *summative evaluation* decides whether to deploy it

-   Mislabeling formative evidence as summative kills credibility

---

## Think-Aloud and RITE

-   *Think-aloud*: "say out loud whatever you are thinking" while doing a concrete task

-   "Given this pull request, use the assistant to summarize the changes"

-   Observe hesitation, backtracking, and expectations that turn out wrong

-   Do not help when they get stuck: you will not be there in production

-   RITE (Rapid Iterative Testing and Evaluation):
    fix the worst problem you observed before the next session [Medlock2002]

---

## Please

-   Ask for evidence before adopting

-   What study supports the claim?

-   Is it applicable to your context?

-   Can you see the raw data?

-   When an executive cites a productivity statistic, ask:

-   Who funded the study?

-   What task was studied?

-   Was there was a control group?

-   Please document and share your organization's experience

---

## Signs of Maturity

1.  "How am I going to be measured in my annual performance review?"

2.  "What do we keep when token costs go up 20X?"

3.  "How much should I spend on insurance, and for what?"

---

## Next Steps

1. Go and look at Bram: <https://judell.github.io/bram/>

2.  Hire a bunch of undergrads and study *them*

-   And share your findings

3.  Hire a law student to put together a half-day workshop on the history of liability

<div class="center">I'd be very happy to help…</div>

---

## Conclusion

-   These mistakes are not unique to AI research

-   The same problems appear in studies of agile, TDD, and other practices

-   The fix is to apply what the human sciences already know

-   So give the job to your UX researchers

-   They already practice the rigorous qualitative and quantitative methods described here

---

<div>
 <div class="center">
 <h2>Thank You</h2>
 </div>
 <div class="row">
 <div class="col-6">
 <img src="../../files/cv/gvwilson-gage-2019.png" width="80%">
 </div>
 <div class="col-6">
 <a href="http://third-bit.com">Greg Wilson</a>
 <a href="mailto:gvwilson@third-bit.com">gvwilson@third-bit.com</a>
 <a href="http://third-bit.com/talks/ai-assess/">http://third-bit.com/talks/ai-assess/</a>
 </div>
 </div>
 <div class="center">
 start where you are · use what you have · help who you can
 </div>
</div>

---

## References

[Bakal2025]
Gal Bakal, Ali Dasdan, Yaniv Katz, Michael Kaufman, and Guy Levin:
"<a href="https://arxiv.org/abs/2501.13282">Experience with GitHub Copilot for Developer Productivity at Zoominfo</a>".
arXiv:2501.13282, 2025.

[Basili1994]
Victor R. Basili, Gianluigi Caldiera, and H. Dieter Rombach:
"The Goal Question Metric Approach".
Encyclopedia of Software Engineering, 1994,
<a class="isbn" href="https://isbnsearch.org/9780471540021">978-0471540021</a>.

[Begel2014]
Andrew Begel and Nachiappan Nagappan:
"<a href="https://doi.org/10.1145/2568225.2568233">Analyze This! 145 Questions for Data Scientists in Software Engineering</a>".
Proc. ICSE'14, 2014,
<a class="doi" href="https://doi.org/10.1145/2568225.2568233">10.1145/2568225.2568233</a>.

[Braun2006]
Virginia Braun and Victoria Clarke:
"<a href="https://doi.org/10.1191/1478088706qp063oa">Using thematic analysis in psychology</a>".
Qualitative Research in Psychology, 3(2), 2006,
<a class="doi" href="https://doi.org/10.1191/1478088706qp063oa">10.1191/1478088706qp063oa</a>.

---

## References

[Braun2019]
Virginia Braun and Victoria Clarke:
"<a href="https://doi.org/10.1080/2159676X.2019.1628806">Reflecting on Reflexive Thematic Analysis</a>".
Qualitative Research in Sport, Exercise and Health, 11(4), 2019,
<a class="doi" href="https://doi.org/10.1080/2159676X.2019.1628806">10.1080/2159676X.2019.1628806</a>.

[Dora2025]
Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, and Sandeep Kumar Shukla:
"<a href="https://arxiv.org/abs/2504.20612">The Hidden Risks of LLM-Generated Web Application Code: A Security-Centric Evaluation of Code Generation Capabilities in Large Language Models</a>".
arXiv:2504.20612, 2025.

[Forsgren2018]
Nicole Forsgren, Jez Humble, and Gene Kim:
Accelerate: The Science of Lean Software and DevOps.
IT Revolution Press, 2018,
<a class="isbn" href="https://isbnsearch.org/9781942788331">978-1942788331</a>.

[Forsgren2021]
Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck, and Jenna Butler:
"<a href="https://doi.org/10.1145/3454122.3454124">The SPACE of Developer Productivity</a>".
ACM Queue, 19(1), 2021,
<a class="doi" href="https://doi.org/10.1145/3454122.3454124">10.1145/3454122.3454124</a>.

---

## References

[Goodhart1984]
Charles Goodhart:
"Problems of Monetary Management: The U.K. Experience".
Inflation, Depression, and Economic Policy in the West, 1984.

[Goodman2012]
Elizabeth Goodman, Mike Kuniavsky, and Andrea Moed:
Observing the User Experience: A Practitioner's Guide to User Research (2nd edition).
Morgan Kaufmann, 2012,
<a class="isbn" href="https://isbnsearch.org/9780123848697">978-0123848697</a>.

[Hall2019]
Erika Hall:
Just Enough Research.
A Book Apart, 2019,
<a class="isbn" href="https://isbnsearch.org/9781952616082">978-1952616082</a>.

[Liang2024]
Jenny T. Liang, Chenyang Yang, and Brad A. Myers:
"<a href="https://doi.org/10.1145/3597503.3608128">A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges</a>".
Proc. ICSE'24, 2024,
<a class="doi" href="https://doi.org/10.1145/3597503.3608128">10.1145/3597503.3608128</a>.

---

## References

[McKinsey2023]
Nora Elsayed, Tarek Elhounsri, and Sven Blumberg:
"<a href="https://www.mckinsey.com/industries/technology-media-and-telecommunications/our-insights/yes-you-can-measure-software-developer-productivity">Yes, You Can Measure Software Developer Productivity</a>".
McKinsey & Company, 2023.

[Medlock2002]
Michael C. Medlock, Dennis Wixon, Mark Terrano, Ramon Ruflair, and Darrell Vaughan:
"Using the RITE Method to Improve Products: A Definition and a Case Study".
Proc. Usability Professionals Association Conference, 2002.

[Meyer2017]
André N. Meyer, Laura E. Barton, Gail C. Murphy, Thomas Zimmermann, and Thomas Fritz:
"<a href="https://doi.org/10.1109/tse.2017.2656886">The Work Life of Developers: Activities, Switches and Perceived Productivity</a>".
IEEE Trans. Software Engineering, 43(12), 2017,
<a class="doi" href="https://doi.org/10.1109/tse.2017.2656886">10.1109/tse.2017.2656886</a>.

[Nielsen1993]
Jakob Nielsen and Thomas K. Landauer:
"<a href="https://doi.org/10.1145/169059.169166">A Mathematical Model of the Finding of Usability Problems</a>".
Proc. INTERACT'93 and CHI'93, 1993,
<a class="doi" href="https://doi.org/10.1145/169059.169166">10.1145/169059.169166</a>.

---

## References

[Pearce2022]
Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri:
"<a href="https://doi.org/10.1109/SP46214.2022.9833571">Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions</a>".
Proc. S&P'22, 2022,
<a class="doi" href="https://doi.org/10.1109/SP46214.2022.9833571">10.1109/SP46214.2022.9833571</a>.

[Peng2023]
Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer:
"<a href="https://doi.org/10.48550/arXiv.2302.06590">The Impact of AI on Developer Productivity: Evidence from GitHub Copilot</a>".
arXiv:2302.06590, 2023,
<a class="doi" href="https://doi.org/10.48550/arXiv.2302.06590">10.48550/arXiv.2302.06590</a>.

[Prechelt2000]
Lutz Prechelt:
"<a href="https://doi.org/10.1109/2.876288">An Empirical Comparison of Seven Programming Languages</a>".
IEEE Computer, 33(10), 2000,
<a class="doi" href="https://doi.org/10.1109/2.876288">10.1109/2.876288</a>.

[Sadowski2019]
Caitlin Sadowski and Thomas Zimmermann (eds.):
Rethinking Productivity in Software Engineering.
Apress, 2019,
<a class="isbn" href="https://isbnsearch.org/9781484242216">978-1484242216</a>.

[Sauro2016]
Jeff Sauro and James R. Lewis:
Quantifying the User Experience: Practical Statistics for User Research (2nd edition).
Morgan Kaufmann, 2016,
<a class="isbn" href="https://isbnsearch.org/9780128023082">978-0128023082</a>.

---

## References

[Stray2026]
Viktoria Stray, Elias Goldmann Brandtzæg, Viggo Tellefsen Wivestad, Astri Barbala, and Nils Brede Moe:
"<a href="https://doi.org/10.24251/hicss.2026.880">Developer Productivity With and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study</a>".
Proc. HICSS'26, 2026,
<a class="doi" href="https://doi.org/10.24251/hicss.2026.880">10.24251/hicss.2026.880</a>.

[Tullis2013]
Thomas S. Tullis and William Albert:
Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics (2nd edition).
Morgan Kaufmann, 2013,
<a class="isbn" href="https://isbnsearch.org/9780124157811">978-0124157811</a>.

[Wicherts2011]
J.M. Wicherts, M. Bakker, and D. Molenaar:
"<a href="https://doi.org/10.1371/journal.pone.0026828">Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results</a>".
PLoS ONE, 6(11), 2011,
<a class="doi" href="https://doi.org/10.1371/journal.pone.0026828">10.1371/journal.pone.0026828</a>.