A Voice from the Back of the Room

Mark Guzdial recently posted another thought-provoking piece, this one about how teachers are biased toward assessing a class's progress by their interactions with its brightest students, and how important it is to assess understanding systematically to find out what's actually going on. We're still wrestling with this in Software Carpentry: what can we put on the web to help students determine whether they have understood a particular topic and are ready to move on?

• Multiple choice quizzes can only be done once: if you don't get the right answers the first time, go back to review the material, then try the quiz again, all it's really testing is your ability to recall that the answer to #3 is 'c'.
• Small programming problems don't work either. Even in simple cases, there are always many right (or at least workable) answers. Novices are, almost by definition, not able to see whether their solution can be mapped onto the instructor's; all they see are the differences, and they come away thinking, "Oh, I didn't get that," rather than, "Huh, I guess there is more than one way to do it." This can be ameliorated a bit by providing sample input and output, so that they can test their programs, but that doesn't help them figure out why theirs is right or wrong.

What else could we do? How else could we help you figure out whether you're ready to move on to the next topic, or ought to review this one again?