I Don't Care Until I Can Check

Over in the Agile Usability group, Larry Constantine writes:

…Capers Jones has been sharing with me some hard data summaries on a variety of development methods and practices gathered from a very large number of projects undertaken by varied organizations that contribute data on bugs, costs, etc., to his company….An interesting thing is that agile methods fare better in most measures, including total cost of ownership of final software product, than practices associated with CMM level 3 but are NOT as good as the Rational Unified Process and all three are trumped by CMM level 5…I don't want to get into the specific numbers (the data set is proprietary anyway)…I want to raise a very different issue: What would it mean to the agile community IF these findings really were valid and true?

To which I can only reply, “Show me the data.” Seriously: if you’re not willing to show people your data and explain where and how it was collected, and how it was analyzed, we should pay exactly as much attention to you as we do to the guy in the bar who claims to have met a guy who met the guy who actually shot JFK. My greatest hope for our upcoming collection on evidence-based software engineering is that it will remind people that neither anecdotes nor trade secrets constitute proof.