I Don’t Care Until I Can Check
Over in the Agile Usability group, Larry Constantine writes:
…Capers Jones has been sharing with me some hard data summaries on a variety of development methods and practices gathered from a very large number of projects undertaken by varied organizations that contribute data on bugs, costs, etc., to his company….An interesting thing is that agile methods fare better in most measures, including total cost of ownership of final software product, than practices associated with CMM level 3 but are NOT as good as the Rational Unified Process and all three are trumped by CMM level 5…I don’t want to get into the specific numbers (the data set is proprietary anyway)…I want to raise a very different issue: What would it mean to the agile community IF these findings really were valid and true?
To which I can only reply, “Show me the data.” Seriously: if you’re not willing to show people your data and explain where and how it was collected, and how it was analyzed, we should pay exactly as much attention to you as we do to the guy in the bar who claims to have met a guy who met the guy who actually shot JFK. My greatest hope for our upcoming collection on evidence-based software engineering is that it will remind people that neither anecdotes nor trade secrets constitute proof.
So did he show you the data?
There is an additional problem with this particular comparison: many measurements are considered an obstacle in agile development, as opposed to CMM (or RUP). If a truly agile team wants to collect them they’ll be handicapped.
There are two ways to work around this, that I can think of: (1) just collect the data that they are producing anyway (bug reports would be an example, but how can we tell that bugs in company A are equivalent to bugs in company B?), even though you’re going to get crappy comparisons because you can’t get the data you really wanted; or (2) collect the data yourself, through an intensive deployment of observers, knowing that you’re introducing serious validity and feasibility problems to the analysis.