Something That Won’t Be On The Final Exam
The final exam for my CSC302 software engineering course was due this week, and I thought I had come up with a good–and fair—question to put on it. Four out of four of the other instructors I spoke to, however, didn’t like it, so I left it out. I’d be very interested in hearing from present and former students (and other teachers) what they think.
This term, I organized the CSC302 students into six teams of about ten people each, and put each team to work on an open source project. Most of the marks in the course are given to the team as a whole, so I wanted a way to assess individual contributions. Having students evaluate their peers would be easy, and in theory would reward people who’ve done more, and penalize people who’ve done less. However, peer evaluation is notoriously vulnerable to gaming and social pressure.
My thought was therefore to measure how well people understood their own contributions, rather than the actual contributions. I thought I could have everyone rate both themselves and all of their teammates. An individual’s grade would then be based on how closely their self-score matched the scores given to them by their teammates. If your teammates all gave you 2/10, for example, and you also gave yourself 2/10, you’d get a high mark for knowing that you’d done a poor job.
Well, you can probably see the flaws, and having thought about my colleagues’ comments, so can I. For one thing, it seems unfair to give people a good grade for doing a poor job. It also seems unfair to grade them twice on the same work (once during term for doing it, and once again at the end for having done it). And it would still be susceptible to gaming, though surprising students with it a final exam would reduce that risk.
All of which raises an interesting question. Peer evaluations are routinely used in industry as part of annual performance reviews; bonuses and promotions routinely depend on what your colleagues think of you. If it’s OK to use them there, why is it so difficult in a course? If anything, there is more incentive to game things in a company where you’re likely to work with people for years than in a course where you’re only working with someone in one or two courses.
What I think is that the way evaluations are done in industry/work place is totally different than the peer reviews that were done in class. In class everybody on the team evaluates each other, and each other is pretty much equal. In industry it’s usually your boss who evaluates you, not your co workers. There isn’t any serious consequence if everybody in the team in class games the system. Now if your boss was evaluating you then s/he’s (usually) under an obligation to evaluate fairly and once your boss makes your evaluation it could also be reviewed by his/her superiors. Your boss would look bad if s/he was caught giving you a good performance review when in actual fact it is not the case.
“If it’s OK to use them there, why is it so difficult in a course?”
Well it’s also OK to use peer evaluations in a course, but in both environments they’re susceptible to abuse. Maybe industry doesn’t care as much?
While things like peer review are used in industry, there are people making a case that traditional employee review techniques are actually harmful. Yes, they provide HR bureaucrats with their beloved documentation, but they don’t actually help staff be more productive. See Esther Derby’s series on “Performance Without Appraisal” http://www.estherderby.com/tag/annual-reviews
If it’s OK to use them there, why is it so difficult in a course?
Maybe this is an example of Paul Graham’s “two kinds of judgement”. The sole purpose (ostensibly) of evaluation in a course is to rank/judge the student accurately. In this context both overestimation and underestimation are serious problems. In a company, it doesn’t necessarily matter if an interview or evaluation is “fair”. For interviews, you want to err on the side of underestimation: you can often afford to reject acceptable candidates but not to hire duds. For evaluations, there are many complex concerns: promoting those who will be accepted in positions of superiority is one that is addressed by peer evaluation.