Descriptive Statistics

Learning Goals

• Compute and interpret mean, median, standard deviation, and percentiles
• Explain when mean and median diverge and what that divergence implies
• Recognize right-skewed distributions and choose appropriate summary statistics
• Use PyPI release-count data to reproduce published descriptive statistics

Mean vs. Median

• The mean is the sum of all values divided by the count; it is sensitive to extreme values
  • One package on PyPI (ccxt) had nearly 11,000 releases at the time of the study
  • That single package pulls the mean release count up to about 11, even though most packages have far fewer
• The median is the middle value in a sorted list; it is not affected by extremes
  • The median PyPI release count is 4
  • A mean of 11 and a median of 4 together tell a story: most packages are small, but a handful of extremely prolific packages distort the average
• When mean and median diverge substantially, report both and explain the gap
  • Reporting only the mean is technically accurate and practically misleading

Variance and Standard Deviation

• Variance is the average of the squared differences from the mean
  • Squaring the differences makes variance sensitive to outliers: one extreme value inflates it dramatically
  • Variance has units of (original units)², which makes it hard to interpret directly
• Standard deviation is the square root of the variance
  • It has the same units as the original data, which makes it interpretable
  • For the PyPI data, the standard deviation is much larger than the mean, which is a sign that the distribution is heavily skewed
• Neither statistic is a good summary for heavily skewed data; use percentiles instead

Percentiles

• The Nth percentile is the value below which N percent of observations fall
  • The 25th percentile (Q1) and 75th percentile (Q3) bound the middle half of the data
  • The 10th and 90th percentiles give a view of the tails without being as extreme as the minimum and maximum
• Percentiles are more informative than min/max for skewed data
  • The maximum PyPI release count is nearly 11,000; the 90th percentile is much lower
  • Min and max are the two values most likely to be unusual, so they tell you the least about the typical case
• The interquartile range (IQR) is Q3 minus Q1
  • It measures the spread of the middle half of the data and is not affected by extreme values
  • The box in a box-and-whisker plot spans the IQR

Skewness

• Skewness measures the asymmetry of a distribution
  • Positive skew (right skew): the tail extends to the right; most values are small, a few are very large
  • Negative skew (left skew): the tail extends to the left; most values are large, a few are very small
• Most software engineering data is positively skewed
  • File sizes, function lengths, bug counts, release counts, response times: all tend to have many small values and a few enormous ones
  • This is not a coincidence; it reflects the fact that these quantities cannot go below zero and have no fixed upper bound
• A right-skewed distribution is one where mean > median
  • The PyPI release count has mean ≈ 11 and median = 4, confirming positive skew

When to Use a Log Scale

• Log scales compress large values and spread small ones, which is exactly what right-skewed data needs
  • Plot the PyPI release counts on a linear scale and you see a spike near zero with a barely-visible tail
  • Plot on a log scale and the shape of the distribution becomes clear
• Any quantity with a long right tail is a candidate for a log scale: file sizes, response times, bug counts, commit counts
• Log-transforming data before computing the mean is common in practice
  • Compute the mean of log(x), then exponentiate: this is the geometric mean
  • For skewed data, the geometric mean is often a better summary than the arithmetic mean

Replication Target

• Load the PyPI release-count dataset and reproduce these published values: mean ≈ 11, median = 4, max ≈ 10,797
• If your numbers differ, check whether the dataset has been filtered or whether packages with zero releases were excluded

"""Descriptive statistics for PyPI release counts."""

import polars as pl

df = pl.read_csv("data/pypi_releases.csv")
counts = df["releases"]
print(f"Mean:   {counts.mean():.1f}")
print(f"Median: {counts.median():.1f}")
print(f"Std:    {counts.std():.1f}")
print(f"Max:    {counts.max()}")
for p in [10, 25, 75, 90]:
    print(f"  {p}th percentile: {counts.quantile(p / 100):.1f}")

Check Understanding

df = pl.read_csv("data/pypi_releases.csv")
spread = df["releases"].variance()
print(f"Spread in releases: {spread:.1f} releases")

spread = df["releases"].std()
print(f"Spread in releases: {spread:.1f} releases")

Exercises