Tidy Data and Polars Basics

Learning Goals

• Explain the three rules of tidy data
• Load, inspect, filter, and select data with Polars
• Handle missing values appropriately
• Write cleaned data back to CSV

Lesson

• Most data you find in the wild is not structured for analysis; cleaning it takes time
  • Tidy data has a simple definition from Wickham [Wickham2014]: each column is one variable, each row is one observation, and one table holds one set of related observations
  • Violating any of these rules makes even simple analyses harder than they need to be
  • Examples of untidy data: a column called jan_sales alongside feb_sales (the month is a variable, not a column name); a row that mixes individual and aggregate data; two different kinds of observation packed into the same table
• Load a CSV with pl.read_csv
  • Polars infers column types from the first batch of rows; check that inference is correct
  • Common surprises: dates read as strings, integers read as floats because one cell is empty, numeric columns that contain notes like "n/a" read as strings
• Inspect a dataframe before doing anything else
  • df.shape returns (rows, columns) as a tuple
  • df.columns lists column names; df.dtypes lists the inferred type of each column
  • df.head() shows the first five rows; df.describe() gives count, mean, std, min, and percentiles for every numeric column
• Select columns with select and filter rows with filter
  • df.select(["col_a", "col_b"]) keeps only those two columns
  • df.filter(pl.col("value") > 0) keeps only rows where the condition is true
  • Polars expressions are lazy by default when using the lazy API; select and filter on a plain dataframe execute immediately
• Null values in Polars represent missing or unknown data
  • A null is not the same as zero; treating it as zero changes the meaning of the data
  • A null is not the same as an empty string; an empty string is present but blank
  • df.drop_nulls("column_name") removes every row where that column is null
  • df.fill_null(value) replaces every null with the given value; use this only when you have a defensible reason for the replacement

• Write a cleaned dataframe back to CSV with df.write_csv("output.csv")

  • Always write to a new file name rather than overwriting the original data
  • If you overwrite the original, you cannot tell whether a problem appeared before or after the cleaning step

• Data availability has its own biases [Acciai2023]

  • Acciai et al. sent data requests to corresponding authors of published papers and found that requesters with Chinese-sounding names received significantly fewer responses than those with Anglo-Saxon-sounding names
  • This means any study that relies on author cooperation to obtain data may have systematically incomplete coverage of researchers from certain regions
  • You cannot fix this bias after the fact; you have to design around it

"""Compute PEP 8 line-length compliance from file data."""

import polars as pl

df = pl.read_csv("data/line_lengths.csv")
print("Shape:", df.shape)
print("Columns:", df.columns)
print(df.head())

clean = df.drop_nulls("line_length")
total = clean["count"].sum()
over79 = clean.filter(pl.col("line_length") > 79)["count"].sum()
print(f"Lines over 79 chars: {over79:,} / {total:,} = {over79 / total:.1%}")
clean.write_csv("data/line_lengths_clean.csv")

Check Understanding

df = pl.read_csv("data.csv")
result = df.filter(pl.col("value") > 0)
result["value"] = result["value"] * 2  # this line

df = pl.read_csv("data.csv")
result = (df
          .filter(pl.col("value") > 0)
          .with_columns((pl.col("value") * 2).alias("value")))

Exercises