From this thread by Mike Hoye, the Engineering Community Manager for Firefox at Mozilla:
- Extremely angry with the state of academic CS research right now.
- We've got a bunch of research papers that have come out of Mozilla's willingness to share a lot of our data. Bugzilla, lots of other stuff.
- We pay attention to that research. We read it carefully, and our operational decision-making is informed by those results.
- But we live and work in an evolving world, so the other thing we do as a matter of course is re-check our data to confirm our assumptions.
- What I want to be able to do is to go back and say, we based this decision in part on the results of paper X. Is that result still valid?
- In a sane world, that's 3 steps.
- Clone the author's VCS
- Run author's [whatever] against an up-to-date dataset.
- Look at the new graph.
- Does the data still support the thesis? Great! Full steam ahead! Has something changed? Pause and think! Either way we're winning.
- But that never happens. Because CS researchers don't publish code or data. They publish LaTeX-templated Word docs as paywalled PDFs.
- And if you care about sciency stuff like "validity" or "reproducibility", even simpleton stuff like "real-world relevance", well, f*** you.
- Let's talk about how far behind schedule we are because people don't share what they know, or didn't make what we learned accessible.
All of which makes me ask yet again: what will it take for researchers publishing in closed-access venues to realize how much damage they're doing to themselves by shutting out people like Mike?
Later: coincidentally, the Journal of Open Source Software has just launched. Like the Journal of Open Research Software, its mission is to give people a place and a way to get academic credit for the tools they build. It would be great to see some software engineering research work appearing in both...