Continuous Installation Checking


This one started with me trying and failing to install some bioinformatics software on my Mac, then turned into a Twitter rant:

But I'm serious. (I'm always serious.) Given how formulaic installation instructions tend to be, could a graduate student or startup build:

  1. an NLP tool to extract actions from installation and configuration instructions,
  2. a monitor to gather traces of the steps taken by actual users (not professional sys admins or package authors) as they try to install the software, and
  3. some sort of Markov chain generator to follow the steps found in #1 while making mistakes similar to those detected by #2?

I don't think this would be easy: the first two parts are each worthy of a thesis (though I suspect the third will be relatively straightforward by comparison). But imagine how useful this would be. Imagine what it would be like if Travis-CI or other continuous integration tools could fire up a lightweight VM and go through your installation instructions every time you committed a change and say, "Nope, sorry, it looks like you've missed a step, and that bit there, the one we've been telling you about for the last three months? Most people are still going to find it confusing, so how about you try explaining it differently?"