Effect Size and Practical Significance

Learning Goals

• Distinguish statistical significance from practical significance
• Compute Cohen's d, common-language effect size, and Cliff's delta and interpret each
• Explain why DeOliveiraNeto et al. found most SE papers fail to report effect sizes
• Use effect size to evaluate TDD outcomes from Fucci et al.

Lesson

• A p-value tells you whether an effect exists; an effect size tells you whether it matters
  • With N = 100,000, even a difference of 0.001 lines of code per hour can produce p < 0.05
  • With N = 10, even a genuinely large difference may not reach significance
  • Statistical significance and practical importance are separate questions that require separate answers
• Cohen's d measures the standardized difference between two group means
  • d = (mean of group A − mean of group B) / pooled standard deviation
  • The pooled standard deviation puts both groups on the same scale
  • Rough guidelines: d ≈ 0.2 is small, d ≈ 0.5 is medium, d ≈ 0.8 is large
  • These guidelines come from Cohen's 1977 textbook and are heuristics, not laws
• Common-language effect size (CLES) is the probability that a randomly chosen value from group A is larger than a randomly chosen value from group B
  • A CLES of 0.5 means the groups are indistinguishable; 0.7 means group A is larger 70% of the time
  • Much easier to explain to a non-statistician than Cohen's d
  • Compute it as the Mann-Whitney U statistic divided by n1 × n2
• Cliff's delta is a non-parametric effect size for ordinal data
  • It measures the probability that a value from group A exceeds a value from group B, minus the probability that a value from group B exceeds a value from group A
  • Ranges from -1 to 1; values near 0 indicate no difference; values near ±1 indicate near-total separation
  • Prefer Cliff's delta over Cohen's d when data is ordinal or when distributions are heavily skewed
• DeOliveiraNeto et al. reviewed more than 5,000 SE papers and found that very few report effect sizes [DeOliveiraNeto2019]
  • Papers report p-values but rarely say how large the effect is
  • This makes it impossible to tell whether a significant result is important or trivial
  • A paper that says "TDD reduced defects significantly (p = 0.03)" but gives no effect size has told you almost nothing useful for deciding whether to adopt TDD
• Furia et al. make a related point: conflicting conclusions in SE often arise from ignoring confounds, not from genuine disagreement about the data [Furia2022]
  • Effect sizes help because they are more stable across analysis choices than p-values
  • Reporting both p-values and effect sizes gives readers more to work with
• The code below loads the Fucci TDD dataset and computes Cohen's d and Cliff's delta for three outcome variables

"""Cohen's d and Cliff's delta for Fucci TDD study outcomes."""

import numpy as np
import polars as pl


def cohens_d(a, b):
    """Compute Cohen's d for two arrays."""
    pooled_std = np.sqrt((a.std() ** 2 + b.std() ** 2) / 2)
    return (a.mean() - b.mean()) / pooled_std


def cliffs_delta(a, b):
    """Compute Cliff's delta for two arrays."""
    n = len(a) * len(b)
    greater = sum(1 for x in a for y in b if x > y)
    less = sum(1 for x in a for y in b if x < y)
    return (greater - less) / n


df = pl.read_csv("data/fucci_tdd.csv")
tdd = df.filter(pl.col("approach") == "TDD")
tld = df.filter(pl.col("approach") == "TLD")

for outcome in ["PROD", "QLTY", "TESTS"]:
    a = tdd[outcome].drop_nulls().to_numpy()
    b = tld[outcome].drop_nulls().to_numpy()
    d = cohens_d(a, b)
    delta = cliffs_delta(a, b)
    print(f"{outcome}: Cohen's d = {d:.3f}, Cliff's delta = {delta:.3f}")

• The paper describes a "small" effect for PROD; verify whether your computed d is consistent with that claim
  • Small means d ≈ 0.2 by Cohen's guidelines
  • Cliff's delta near 0 means the groups overlap substantially
  • A "statistically significant small effect" in a software engineering study should change practice only if the practical context makes even a small improvement worthwhile

Check Understanding

def cohens_d(group_a, group_b):
    return (group_a.mean() - group_b.mean()) / group_a.std()

def cohens_d(group_a, group_b):
    pooled_std = np.sqrt((group_a.std()**2 + group_b.std()**2) / 2)
    return (group_a.mean() - group_b.mean()) / pooled_std

Exercises