From this thread by Mike Hoye, the Engineering Community Manager for Firefox at Mozilla:

  • Extremely angry with the state of academic CS research right now.
  • We've got a bunch of research papers that have come out of Mozilla's willingness to share a lot of our data. Bugzilla, lots of other stuff.
  • We pay attention to that research. We read it carefully, and our operational decision-making is informed by those results.
  • But we live and work in an evolving world, so the other thing we do as a matter of course is re-check our data to confirm our assumptions.
  • What I want to be able to do is to go back and say, we based this decision in part on the results of paper X. Is that result still valid?
  • In a sane world, that's 3 steps.
    • Clone the author's VCS
    • Run author's [whatever] against an up-to-date dataset.
    • Look at the new graph.
  • Does the data still support the thesis? Great! Full steam ahead! Has something changed? Pause and think! Either way we're winning.
  • But that never happens. Because CS researchers don't publish code or data. They publish LaTeX-templated Word docs as paywalled PDFs.
  • And if you care about sciency stuff like "validity" or "reproducibility", even simpleton stuff like "real-world relevance", well, f*** you.
  • Let's talk about how far behind schedule we are because people don't share what they know, or didn't make what we learned accessible.

All of which makes me ask yet again: what will it take for researchers publishing in closed-access venues to realize how much damage they're doing to themselves by shutting out people like Mike?

Later: coincidentally, the Journal of Open Source Software has just launched. Like the Journal of Open Research Software, its mission is to give people a place and a way to get academic credit for the tools they build. It would be great to see some software engineering research work appearing in both...

This post originally appeared at It Will Never Work in Theory.