A Base Case for Empirical Software Engineering Research

I've been saying for years that programmers ought to pay more attention to empirical studies of software engineering and base their practices on evidence rather than strong opinion. I was challenged on this yesterday when someone asked me to cite studies showing that a bug tracker is a better way to manage backlog than a shared spreadsheet. I couldn't, and still can't.

I also can't find any studies showing that version control is a better way to manage software projects than mailing files around or dumping them in a shared folder. I "know" it's true—I wouldn't work on a project that didn't use version control—but then again, my aunt "knew" that putting colored crystals on her chi points would relieve her arthritis. As far as I can tell from outside the Great Paywall of Academia, nobody's ever actually done the study.

That's kind of embarrassing, but it's also an opportunity. The biggest open problem in empirical software engineering research is measuring productivity: lines of code per hour and story points per sprint are easy but meaningless, and we haven't agreed on anything more sophisticated.

So here's my proposal: let's use version control as a filter for measures of programmer productivity. If metric X doesn't show that groups using version control outperform groups that don't, I think we can safely discard it. More usefully, how different metrics measure those groups' productivity differences would allow us to compare those metrics more directly. It would also be a good starter exercise for the better software engineering course I've been musing about.

Just a thought...

comments powered by Disqus