Something That Won't Be On The Final Exam

The final exam for my CSC302 software engineering course was due this week, and I thought I had come up with a good–and fair—question to put on it. Four out of four of the other instructors I spoke to, however, didn’t like it, so I left it out. I’d be very interested in hearing from present and former students (and other teachers) what they think.

This term, I organized the CSC302 students into six teams of about ten people each, and put each team to work on an open source project. Most of the marks in the course are given to the team as a whole, so I wanted a way to assess individual contributions. Having students evaluate their peers would be easy, and in theory would reward people who’ve done more, and penalize people who’ve done less. However, peer evaluation is notoriously vulnerable to gaming and social pressure.

My thought was therefore to measure how well people understood their own contributions, rather than the actual contributions. I thought I could have everyone rate both themselves and all of their teammates. An individual’s grade would then be based on how closely their self-score matched the scores given to them by their teammates. If your teammates all gave you 2/10, for example, and you also gave yourself 2/10, you’d get a high mark for knowing that you’d done a poor job.

Well, you can probably see the flaws, and having thought about my colleagues’ comments, so can I. For one thing, it seems unfair to give people a good grade for doing a poor job. It also seems unfair to grade them twice on the same work (once during term for doing it, and once again at the end for having done it). And it would still be susceptible to gaming, though surprising students with it a final exam would reduce that risk.

All of which raises an interesting question. Peer evaluations are routinely used in industry as part of annual performance reviews; bonuses and promotions routinely depend on what your colleagues think of you. If it’s OK to use them there, why is it so difficult in a course? If anything, there is more incentive to game things in a company where you’re likely to work with people for years than in a course where you’re only working with someone in one or two courses.