Data Dictionaries

I was helping some friends analyze some data today, and discovered that the ./data directory in the project they had inherited contained a file called manifest.csv that was loaded and echoed in the top of their analysis notebook. I can’t show you what it contained—their data isn’t public—but the equivalent for Allison Horst’s Palmer Penguins dataset would look something like this:

penguins,species,text,NA,false,common name of species
penguins,island,text,NA,false,island where data collected
penguins,bill_length,number,mm,true,bill length (Figure 1)
penguins,bill_depth,number,mm,true,bill depth (Figure 1)
penguins,flipper_length,number,mm,true,flipper length (Figure 2)
penguins,body_mass_g,number,mm,true,bird weight
penguins,sex,text,NA,true,bird sex

It’s easier to see and appreciate laid out like this:

table column type unit na meaning
penguins species text NA false common name of species
penguins island text NA false island where data collected
penguins bill_length number mm true bill length (Figure 1)
penguins bill_depth number mm true bill depth (Figure 1)
penguins flipper_length number mm true flipper length (Figure 2)
penguins body_mass_g number mm true bird weight
penguins sex text NA true bird sex

The table name is included because the manifest.csv I’m imitating described several related data files; one of the column descriptions even said, “Foreign key into other_table/other_name”.

This doesn’t include everything—for example, it doesn’t specify which text fields are enumerations (or factors, if you’re a statistician)—and the figures referred to in the original manifest.csv aren’t anywhere in the project repository—but wouldn’t life be better if every project you worked with came with something like this? Having once spent several days trying to figure out which temperature measurements in a dataset were °C and which were °F, having SI units somewhere discoverable was enough to make me swoon.

In the wake of posts about Shopify's support for white nationalists and DataCamp's attempts to cover up sexual harassment
I have had to disable comments on this blog. Please email me if you'd like to get in touch.