CSC49X: Figuring Out Where the Goalposts Are

Another batch of CSC49X projects are due to start next week, so this seems like a good time to fulfil the promise I made two years ago to write up how the course is run, and why. (For readers outside U of T, CSC494/495 is a term-long project course, in which students work directly under the supervision of a professor.) Step 1: Figure out where the goalposts are, so you know which way to kick the ball. If you’re an academic, this means finding something that is interesting and challenging enough to be publishable, but not intractable. In a startup, it means figuring out what you can build that people will pay for. Working for someone else? Check your job description (including criteria for performance evaluation and bonuses).

Life’s easier if you’re a student: all you have to do is check the marking scheme. Here’s what we’ll be using in 49X this fall, and why.

Warmup exercise: 10%
The purpose of the warmup exercise is to give students a chance to familiarize themselves with the tools they'll be using in the rest of the course. Whether it's Hibernate, , Flash, or something more exotic, students have to have some hands-on experience before they can start planning the rest of their term. The code students write for this is thrown away after being graded; they are strongly encouraged not to try to turn their finger exercises into finished products.
Analysis & estimation: 10%
Now that they know what they're going to be working with, students can analyze their customers' requirements, figure out what features will satisfy them, prioritize, and draw up a schedule. This is an important enough topic to merit a post of its own; until then, check out the lecture on development at the Software Carpentry site.
Code: 10%
Yes, that's right: the code students write for this course is only worth 10% of their final grade, even though it's where most of them will spend the bulk of their time. There are two reasons for this: (1) if they don't know how to program, they shouldn't be in 49X, and (2) if they don't create some code, they can't test, do a demo, or write their final report.
Testing: 10%
Testing is just as important as coding, so it's given the same weight. Note, though, that only automated tests count: if I can't check the project out of Subversion and re-run the tests (possibly after configuring a server or two), then as far as I'm concerned, the code hasn't been tested. And it's no good saying, "But I can't write unit tests for my GUI," because (a) if you design your program the right way, you can write 'em for the back end, and (b) tools like Selenium can do a lot more than most students realize is possible.
Demos: 10%
I used to require students to prepare a 20-minute PowerPoint lecture on a topic of their choosing, and deliver it to either their coursemates or a junior class. It was a valuable experience, but it ate up a lot of time. This summer, I took a page from David Crow's DemoCamp playbook, and had students do 10-minute demos instead. It worked well: in most cases, the first one (which isn't graded) was shaky, but the second (which is) was slick, so I'll stick to this format in future.
Teamwork: 10%
Most of the students I take on are mature, responsible, and polite; the teamwork mark is the stick I use to beat the ones who aren't. Everyone starts with 10 out of 10, but if they only do their work at the last moment, or check in code that breaks the build, or are rude or condescending in email, marks come off.
Final report: 40%
Students deliver software to their customers; they deliver their final reports to me. Each term, I tell them to write the document they wish they'd had when they started the project, so that whoever inherits their work will have an easier life. What does it do? Why does it do it that way? How is it structured? Built? Tested? What went well? What should the next person or team avoid like stinky cheese? Most importantly (for me, anyway), what have the students learned from the whole experience? Past reports have ranged from 8 pages to almost 50; these days, I look for 8-10 pages per student (i.e., roughly 25 pages for a 3-person team). If time permits (i.e., if students start work early enough), they can submit their report once to get feedback from me, revise it, and then resubmit for marking. This usually makes at least one full letter grade difference in the mark, which gives teams yet another reason to plan their time carefully.

This grading scheme is labor-intensive: I probably spend 6-8 hours reading and grading each project in a term, or roughly 50 hours in total. I’ve thought several times about using peer grading to reduce my load (and give students some experience of what life is like on the other side of the red pen), but I’ve never been able to convince myself that it would actually work.