How Do You Determine the Health of a Software Development Project?
This one has come up before, but with a near year dawning, I think it’s worth revisiting. Suppose you had been asked to audit a medium-sized software development project—something with a couple of dozen people, split between two or three sites, with a couple of hundred thousand lines of moderately complex code already written. What would you do to determine how healthy the it was? Obvious things include:
- Ask the team how positive they feel about what they’re doing. (Happy teams aren’t always productive, but unhappy teams almost never are.)
- Compare their actual development practices against some kind of checklist. (Version control? Yup. Test-driven development? No, but most people still don’t actually do that. Some testing, with continuous build on the back-end, is a more reasonable expectation…)
- Read their code. In my experience, this is a lot more work than the previous two options, primarily because most projects don’t have a useful architectural overview.
- Help them out for a few days (e.g., do some testing).
What else? What would you do, and why?
Now put your answer aside, and answer a slightly different question. What would you do if you were faced with a dozen teams, each with four to six people and roughly ten thousand lines of code? This is the situation hundreds of lecturers like me face every term when teaching project-based software engineering classes to undergraduates, and all of the strategies above fail:
- By definition, student teams are composed of novices who don’t (yet) know how to work productively in teams, so asking them how they think they’re doing is sort of like asking a random number generator.
- Comparing their development practices against a checklist fails because you either publish the checklist or you don’t. If you do, students work to that whether they believe in it or not. If you don’t, you’re playing “I’ve got a secret and I won’t tell”, which doesn’t do a lot for morale.
- Reading their code has the same problem; it also measures what they produce, rather than how they produce it, and the latter is really what we ought to be teaching.
- Helping them out for a few days (or sitting in on team meetings) would be the best method, but there simply aren’t enough hours in the week.
Pious hopes and protestations notwithstanding, students mostly learn what’s assessed. This is partly an economic decision (given a choice between putting hours into something that will show up on their transcript, and something that won’t, the former is the rational strategy), but it’s also true that if we grade one thing and not another, we’re sending a pretty clear signal that we think the first is more important than the second. I therefore believe that if we can’t come up with robust, affordable ways to give students marks for how they build software, we will continue to produce graduates who know how to write legal Java or Python, but not how to build real-world applications without heroic effort. I’d welcome your proposals…
It seems reasonable to use a “checklist” based on concepts taught earlier in the semester (or even concurrently). You’re showing them techniques like using version control and TDD for a reason. If they pick them up for use in their own project without being explicitly told they will be graded on their use, that shows that not only did they learn the technique but they see the benefit. You probably shouldn’t take points away for not using them (esp. if they are introduced concurrently with the project), but add points for each technique adopted.
I assume the projects are customer driven, rather than simply make-work tasks? Customer satisfaction should play a role in the evaluation. Did they meet the requirements? Were they on time? Did they cut appropriate features (in consultation with the customer) to make a deadline if they *weren’t* on schedule at some point?
Self-evaluation is important, too, even though it is hard for newbies to be objective (as you point out). Identifying their mistakes and identifying actions they could have taken to avoid them, even after the fact, shows that they have learned something. How could the team have worked better together? Which of the practices that they learned about in class might have made the project more successful?
It depends on what kind of “health” you’re talking about. Specifically:
If you mean “a team who is delivering salable product,” then I’m more interested in how they’re meeting deadlines and producing shippable (meaning: enough features and few enough bugs to satisfy customers) software than I am about the state of the codebase.
If you mean “a team making maintainable, solid code,” then code reviews are probably the best thing you can do. Is the code readable, is the documentation adequate, are there unit tests, do the various team members generally understand all the code and where things are.
It matters because there’s beautiful code (sorry for the reference!) that doesn’t the get job done or isn’t delivering features quick enough, and there’s successful commercial teams with less than stellar code but who are doing exactly what they need to be doing.
You could argue that sacrificing code “goodness” for the sake of delivering features isn’t wise in the long run, and I would agree if the software is supposed to be in a mature place in its lifecycle, but if it’s new software the equation isn’t the same and it might indeed be the correct trade-off.
You might have asked them to “keep something running” from half-way through their time so you have something to test?
- Paddy.
Well, evaluating teams’ software development environment (and then recommending and implementing changes) is basically how I make my living. So I guess I should contribute something
What I do is get a quick understanding of the products, then start asking questions. I use the chapters of “Practical Development Environments” as a guide, since they were written with this in mind. For example, version control – number of developers, number of sites, time for checkout for developer, release builds, number of branches, branching and merging strategies, etc. Then on to the build system, testing, documentation, automation, communication. You get the idea. It usually takes about an hour for 2 to 4 people. There are other more specific questions depending on the kind of product (desktop, web) and domain (networking, libraries). Then I summarize my understanding and email it to the potential client, usually with some recommendations, and we go from there.
However, it’s only once I’ve looked at source code, build files, build breakages, test coverage, the release process and internal documentation that I start to get a feel for the health of a project. Some things are very subtle – who are the technical leads, do they work well with each other, how does the rest of the team view their direction.
Now, to answer your real question. How would I evaluate teams of people learning to develop software together? I’m going to assume that the projects are a few months in duration and that guidance in the form of the previous year’s best projects and your comments on them is available. I’d want to see evidence of thought in how the project’s code is structured, and appropriate comments are important to everyone involved. Tracking bugs is a good sign, along with their resolution. Unit testing the harder parts and regression tests are excellent. Writing a good user guide or online help shows good taste.
I guess I like the idea of half the points for getting there and half for how you got there, but no-one learns in a vacuum so much of this depends on how the examples of good development are presented.