As Mark Guzdial recently reported, Allison Tew has successfully defended her PhD thesis at Georgia Tech. For her thesis, she constructed and validated a language-independent measure of CS-1 knowledge. I’ll post again when her test becomes available—I can’t wait to see it (and use it in Software Carpentry).
Jen Dodd recently posted an article titled “3 rules for running events“, plus one metarule that I particularly appreciated: “Stop deluding yourself.” In the same spirit, I’d like to offer up three rules for running student programming projects. To set the stage, here’s the number of student programming projects I’ve organized, supervised, or otherwise been guilty of since David Wallace first asked me to look after a couple of summer interns in Edinburgh half a lifetime ago:
| Year | 1987 | 1988 | 1989 | 1990 | 1991 | 2002 | 2003 | 2004 | 2005 | 2006 | 2007 | 2008 | 2009 | 2010 |
| Number | 2 | 4 | 12 | 14 | 26 | 7 | 13 | 35 | 42 | 34 | 38 | 78 | 110 | 49 |
Yes, the numbers for 2008 and 2009 are crazy, but those are the years I ran consulting projects at the University of Toronto and started UCOSP. If you only count students I directly supervised, the numbers for 2008-09 drop back down to the high thirties—say, a dozen or so per term, three terms a year.
So what have I learned in those 23 years?
This was the hardest one for me to learn, and it’s almost always the hardest to get across to both students and their clients. University terms may be thirteen weeks long, but students are usually juggling five courses, and many have part-time jobs as well. That means they can only put eight hours a week into their project without sacrificing grades somewhere else. If you figure a full-time work week is 35 hours, that means students actually spend 8×13/35 = a bit less than three weeks working for you. In that time, they have to:
That’s an awful lot to squeeze into three weeks: very few open source projects expect their GSoC students to start checking things in after three weeks of full-time work, but students in school are expected to be done in that time. Prof. Karen Reid says that she usually divides the term into three pieces:
I find that I spend the first 3 weeks working hard to get the students up to speed and essentially demanding that they get something real done in the first 3 weeks. In other words, my students are more successful if they push hard at the beginning. After that, they usually have a good idea of what they need to do for the remainder of the term and I can kind of let them set the pace. Then I spend the last 3 weeks defining what it means to be done.
There’s another catch lurking in here too. The iron triangle of project management is scope, schedule, and resources. In a student project, both the schedule and resources are fixed (13 weeks and N students respectively), so the only thing that can give is scope. There are two ways to reduce it: lower quality, or fewer features. Lowering quality is self-defeating—the students you want in a project course are the ones who take pride in their work and care about their grades (which aren’t necessarily the same thing), and they’re not going to like being told that the only way to pass a course is to produce crap.
That leaves the number and scope of features as the only free variable. Problem is, neither students nor clients are going to be excited about fixing a couple of minor bugs or adding one small new feature. If you want to get people on board, you have to aim higher, and be willing and able to reduce scope as the term goes on without making anyone feel like the project has failed—which brings us neatly to our second rule.
It really isn’t. When I ask students I’ve supervised in the past what they learned in their project, they never mention technology—never. They might have learned Ruby on Rails, or CUDA, or touch-screen interface design, or database performance optimization, but that’s not what they remember afterward. What sticks is how to run a project: how to run a progress meeting, review someone else’s code, manage their time, present their work in five minutes or less, and negotiate scope with a client.
I’ve tried teaching these things in regular software engineering classes, but it has never worked. (This is one of the reasons I have so little use for standard undergrad software engineering textbooks: you can talk about riding bicycles all you want, but the only way to learn how to do it is to do it.) On the upside, once I students understand that I’m trying to teach them process, rather than technology, the problems I mentioned in the previous section are greatly reduced: cutting the set of features we’re going to deliver, for example, becomes an exercise in scope negotiation rather than a failure on the students’ part.
So what goes into a rational student-oriented development process?
And that’s about it. On some projects, I’ll ask students to draw up a plan for the term at the end of their second or third week (i.e., once they’ve learned something about the problem—if they have to do it at the start of term, waterfall-style, all they can do is write some science fiction and hope I won’t hold them to it). On others, there’s some formality around handing off their code to their client, such as submitting it as a patch, doing a presentation at the client site, or showing off their work to all comers at a local pub.
Other people handle process differently, of course. Andrew Ross, of Ingres, says:
I tend not to have regular weekly meetings with my teams. Instead, we have meetings as needed to discuss things that can’t be covered acceptably in emails/IM’s/IRC/calls. We do the latter constantly. The more important underlying concept is keeping students from drifting away and losing contact.
I once had three students working on separate projects during the same summer term. Two had straight A’s; the third was struggling to maintain a low ‘B’ average, but he’s the only one I would have hired back, because he was the only one I could actually rely on. One of the ‘A’ students had spent his whole life acing exams, and didn’t know how to do anything else. He panicked when asked, “What do you think we should do next?” Literally—you could see his pulse race and his mouth dry out. The second had the same fatal flaw I had when I was twenty: he’d do the first three quarters of every job in record time, but getting the next 20% out of him was like pulling teeth, and the last 5% never got done all.
The third student, though, was as reliable as a grilled cheese sandwich. If he told me on Monday that something was going to be done on Friday, it was done on Friday; when I asked him, “Where are you?” he always gave me a straight answer: no “almost done”, no “just another couple of bugs” if he hadn’t actually started. It took me a couple of months to appreciate him, but once I did, I started looking for that same quality in every student I interviewed.
Of course, this isn’t to say that every student with low grades is a gem waiting to be uncovered, or that everyone with an ‘A’ average is unreliable. Far from it: grades are a fairly reliable indicator of ability and persistence, especially grades in courses that no one loves. But the correlation is a lot weaker than I, a former ‘A’ student, once believed.
Keep in mind that even the steadiest students will doubt themselves sometimes. Quoting Karen Reid again:
I find I spend a lot of time reassuring students who are climbing the learning curve. Having different levels of expectations for students depending on their background is something I have to explain to students used to the same evaluation standards.
And “steady beats smart” applies to supervisors as well as students. If you’re unreliable—if you miss meetings, promise to do things but don’t get around to them, or pretend to know more about technical matters than you actually do—your students will respond in kind. If you can’t or don’t commit at least 3-4 high-quality hours a week for each project you’re running, it would be better for everyone if you did something else. (This is, by the way, one of the many reasons I prefer team projects to individual ones: the number of hours required per project grows only slowly with the team size, at least up to half a dozen students, so you can reach more students without sacrificing everything else.)
And finally, a metarule:
Students won’t ever enjoy a project more than you do. After all, they have to do all the hard work, like tracking down bugs, while you get to do the fun stuff like argue over what it’s all supposed to do. And if you’re not having fun, they will quickly start to treat the project like just another course. It’s very hard to pull out of that downward spiral, so don’t get into it: no matter what happens, grit your teeth and have some fun. Go out for ice cream; borrow a projector and introduce them to Tron, WarGames, or Startup.com. They’ll remember that long after the course is over, too, and so will you.
Later: a recent study confirmed what most of us probably knew already: what makes people happiest (or saddest) are group events and achievements, not individual accomplishments. Maybe that’s why students enjoy team projects, and come away appreciating most what they learned about teamwork rather than technology…
I just set up a new 64-bit Windows 7 machine at work; here’s what I installed (in order):
I’m sure there’s more to come, but I can now do a day’s work without having to download and install anything else.
Another good post from Mark Guzdial, this one reporting on a CS education conference he’s attending. Among the talks was one summarizing seven non-myths about computing, i.e., things that people sometimes claim are myths, but which are actually true:
Cecilia d’Oliveira and colleagues recently wrote an essay in Science about MIT’s OpenCourseWare initiative, ten years after its inception. Among the stats:
OCW currently receives upwards of 1.5 million visits each month from
900,000 unique individuals. Students have grown to 42% of the audience, and educators and independent learners now constitute 9% and 43% of visitors, respectively. Twelve percent of educators responding to a March 2010 visitor survey indicated that they do incorporate OCW materials into their own content as anticipated, but educators more frequently use OCW for personal learning (37%), to adopt new teaching methods (18%), and as a reference for their students (16%). Students were largely expected to use the site as a supplement to materials they received in their own classes, a use identified by 40% of students. Just over 43%, however, indicated that they also use OCW for personal learning beyond the scope of their formal studies, and a further 12% use it as an aid in planning their course of study. Independent learners use OCW in a variety of personal (41%) and professional (50%) contexts, including home-schooling children and keeping up on developments in their professional field. 66% of visitors indicate they are mostly or completely successful at meeting their educational goals for visiting the site.
Mike Conley has posted some early results from his study of student code reviews. One of the most interesting is that students who have reviewed their peers’ code believe they know more about the quality of their own work. He should be finished his analysis in a few weeks—I’m looking forward to hearing what else he’s found.
Andrew Smith, a graduate student I have been working with, is looking for people to take part in a study:
…to determine what advantages and disadvantages three types of code review tools have.
As a participant you would spend about an hour doing 3 code reviews (20 minutes each):
- One on paper, just like marking in the good old days.
- One using ReviewBoard, a popular code review tool.
- One using a Tablet PC, with custom-built software.
No code review or marking experience is required, all that you need is to be familiar with either C or Java.
If you can spare an hour, we’d both be grateful for your help.
Jorge Aranda successfully defended his PhD thesis this morning. It’s the culmination of several years of thought-provoking work; I look forward to more great things in the future.
A couple of days ago, I asked a question about generating tests for Nose. Jacob Kaplan-Moss (of Django fame) provided a elegant answer that I can understand in retrospect, but would probably not have been able to come up with myself. If you’re interested in Python decorators and generators, check out my summary on the Software Carpentry blog.
Recent Comments