Reading Studies Critically

In 2023, McKinsey published "Yes, You Can Measure Software Developer Productivity" [McKinsey2023]. It proposed a framework involving dozens of metrics across multiple dimensions and was widely cited by managers and executives as justification for measuring individual developer output. Researchers found the paper's methodology thin and its claims overclaimed. The paper was not obviously fraudulent—it was something more insidious: confident, plausible, and wrong in ways that non-specialists could not easily detect. Kent Beck, one of the authors of the Agile Manifesto.

The gap between belief and evidence runs deep. [Devanbu2016] studied developers at Microsoft and found that beliefs are strong, but formed from personal experience rather than research, and do not reliably correspond to actual data from the projects those developers work on. This session gives you the critical reading skills needed to do better.

How to Read a Research Paper

• Start with the abstract: what claim is being made?
• Jump to the methods before reading the results: given this design, what can this study actually establish?
• Read the results section: do the numbers support the claims made in the abstract?
• Read the discussion and limitations: what do the authors say they cannot conclude? What do they not say?
• Check the references: are the cited sources used accurately?
• An efficient order: abstract → methods → limitations → results → introduction (the introduction is usually the last thing written and the most polished, not the most informative)

HARKing

• HARKing stands for Hypothesizing After Results are Known
• A researcher runs an analysis, finds an interesting pattern, and then writes the paper as if that pattern was predicted in advance
• HARKing inflates false positive rates because exploratory analysis is presented as confirmatory analysis
• Pre-registration commits hypotheses and analysis plans before data collection, making HARKing visible
• Signs of HARKing
  • Vague hypotheses in the introduction that are suddenly very specific in the results
  • An implausibly clean story with no unexpected findings
  • No pre-registration

p-Hacking

• p-hacking means trying multiple analyses until you get p < 0.05, then reporting only that analysis
  • Also called "data dredging," "specification searching," or "researcher degrees of freedom"
• Usually done by trying:
  • different subsets of the data
  • different covariates
  • different outcome measures
  • different time windows
• With 20 independent tests at p < 0.05, you expect one false positive by chance
• Signs of p-hacking:
  • Many conditions with exactly one that is significant
  • Results that depend on excluding a small number of observations
  • No statistical correction for doing multiple comparisons

Publication Bias

• Publication bias: journals publish positive results (significant effects) more often than null results
• Researchers internalize this and are less likely to submit null results (the file drawer problem)
• Consequence: the published literature overestimates effect sizes and overstates how often phenomena replicate
• Meta-analysis tries to combine results across studies
• Funnel plots can reveal asymmetry suggesting publication bias
• Pre-registration and registered reports (where journals commit to publish regardless of outcome) are partial remedies

Conflicts of Interest

• Industry-funded studies in software engineering often produce results favorable to their funders
  • Not through fraud per se, but through study design choices, outcome selection, and framing
• The GitHub Copilot study [Peng2023] was conducted by GitHub employees and published on arXiv without peer review before appearing in venues
• Conflicts of interest should be disclosed
  • Their absence in a paper is a yellow flag
• Ask who benefits if this finding is true? Who paid for the study? Who employs the authors?

Operationalization as a Site of Flexibility

• A subtler form of researcher flexibility than p-hacking is operationalization: how an abstract concept like "productivity" is converted into a number
• [Prechelt2000] showed that the same underlying data produced ratios of 105x, 17x, 11x, or 5x depending on which operationalization was used
• An operationalization chosen after seeing the data is functionally equivalent to p-hacking
  • The analysis looks confirmatory but was actually exploratory
• Signs of post-hoc operationalization
  • Vague outcome definitions in the introduction that become very specific in the results
  • Multiple outcome measures reported with emphasis on the most favorable one
  • No pre-specified primary outcome
• The same trap appears in ML research: Risse et al. found that 90% of machine learning vulnerability detection papers define the problem as function-level binary classification, yet context is almost always required to make a real vulnerability call [Risse2025]
  • High benchmark scores arise from spurious correlations in the data
  • The benchmark measures what is easy to operationalize, not what practitioners actually need
  • This is the measurement-of-the-easy-half problem applied to an entire research subfield

A Checklist for Evaluating an Empirical SE Study

• Organizing questions around the four validity categories makes it easier to identify where a paper is weakest [Campbell1963, Wohlin2000, Juristo2001]
• Conclusion validity:
  • Was the sample large enough to detect an effect of the expected size?
  • Are effect sizes reported alongside p-values?
  • Were multiple comparisons corrected for?
• Internal validity:
  • Was there a control group or a credible counterfactual?
  • Were participants randomly assigned to conditions?
  • Could learning effects, novelty effects, or boredom explain the result?
  • Are confounding variables identified and addressed?
• Construct validity:
  • Is the operationalization of key concepts explicit and defensible?
  • Does the measurement actually capture what the claim is about?
• External validity:
  • Who are the subjects, and are they representative of the population the conclusions address?
  • How closely does the task resemble real-world work?
  • Are the conclusions proportionate to the evidence?
• Reporting (needed for replication and honest interpretation):
  • Are the study's factors defined precisely enough to reproduce it?
  • Is the analysis plan described (not just the analysis that was run)?
  • Are limitations discussed honestly?
  • Is there a conflict of interest disclosure?
  • Has the study been replicated or pre-registered?

When the Outcome Measure Is a Dashboard

• Many software engineering studies use dashboard data as their outcome
  • Commit counts, burndown charts, ticket closure rates, deployment frequency
• Dashboards embed value judgments and carry systematic risks
  • They favor numbers over narrative, miss context, invite Goodhart's Law gaming, have data quality problems, and capture only what was explicitly tracked [Sadowski2019]
• When evaluating a study that used dashboard data, ask:
  • Could the metric have been gamed once participants knew it was being measured?
  • Is the outcome actually visible in this data source, or is the data a proxy for a proxy?

Misconceptions

Peer review detects p-hacking and HARKing.

Reviewers see the final manuscript, not the analysis history. Without access to pre-registered protocols or raw data, they cannot distinguish a cleanly hypothesized analysis from one that was tidied up after the fact.

Pre-registration guarantees that a study's results are valid.

It reduces one class of problem—undisclosed flexibility in analysis—by making deviations visible. It does not fix flawed designs, bad measurements, unrepresentative samples, or errors in execution.

Industry-funded research is always biased toward favorable results.

"Always" is too strong: some industry-funded studies are rigorous and honest. The concern is systematic, not universal: on average, funding source predicts outcome in ways that independent replication often does not support.

A well-cited paper is probably right. Citation counts reflect influence, not accuracy.

Papers get cited because they tell a useful or interesting story, whether or not the story holds up to scrutiny. Many highly-cited results in software engineering have failed to replicate.

Check Understanding

Exercises