October 26, 2012: Counting to Five (or, A Plan for Online Tutorials and What's Wrong With It)

My daughter is five and a half years old, and some time in the last couple of weeks, she made a small cognitive breakthrough. It used to be that if I asked her, "What's five plus three?", she would carefully count five fingers, then count three, then go back and count them all together to get eight. She doesn't do that any more; instead, she holds up five fingers, then counts, "Six... seven... eight!" to get the answer. It may sound like a small thing, but it's not: the idea that five is always five, even if you didn't count it this time around, is such a big one that we forget it ever had to be learned.

A lot of my time these days goes into trying to count from five in online learning. While an ignorance of prior art might be considered an asset in Silicon Valley, I have enough failures and harder-than-it-should-have-beens behind me to make me want to build on what people have done before rather than reinvent things, even if the latter is more fun. What kinds of distance learning have worked in the past? How many of those ideas still work online? What new ideas work? What doesn't work, and why not? And how do we know? I don't believe for a moment that all the answers I want are already out there, but I'd have to be pretty arrogant to believe that none of them are, or that it'd be faster for me to rediscover things than read about them.

For example: between March and July we ran two dozen online tutorials for people who'd been through Software Carpentry workshops. We used BlueJeans (and later Vidyo) to share the presenter's desktop with the learners, and two-way audio (later supplemented with IRC-style text chat) both for questions directed at the presenter, and sideways Q&A between learners. The net effect was exciting and frustrating in equal measure: exciting because we could actually do it, and frustrating because the technology didn't work for some people, and the experience was never really satisfying for anyone. Even when learners were sitting together in small groups so that they could have high-bandwidth interactions with each other during the tutorial, the result was a pale imitation of being there in person.

At the same time, though, I was helping people one-to-one, and that seemed to work a lot better. These sessions typically started with a "can you please help me?" email, then turned into a Skype call and some desktop sharing. I found was that diagnosing a novice's problem was a lot easier when I could see what they were doing (or when they could see what I was doing—we did it both ways). The reason was that almost by definition, novices don't know how to explain their actual problem: if they knew enough to say, "I think my EDITOR variable must be set incorrectly," they'd almost certainly be able to fix the problem themselves. Email back and forth with someone who's still trying to get from A to B is a lot like a game of Twenty Questions: can you run this command and send me its output, have you edited that file, and so on. A few seconds of interaction short-circuited a lot of that, which made it much more satisfying on both sides.

That experience suggests a different kind of online teaching than the MESS (Massively Enhanced Sage on the Stage) being used by Coursera, Udacity, the Khan Academy, and so on. Grossly misrepresented for rhetorical purposes, their model is:

  1. Watch a recording of a sage explaining something in a single fixed way.
  2. Try to do something that the sage decided would be good for you.
  3. If it doesn't work, hope that the static hints embedded in the robograder trigger a moment of satori.
  4. If not, try to translate your incomplete and imperfect understanding of what you're actually doing and why it isn't working into a coherent explanation.
  5. Hope that someone reads it, that you happened to include enough of the right information for them to diagnose your actual problem, and that they are able to write an explanation that you can understand given your current state of knowledge.
  6. Repeat.

Minus steps 1 and 2, this is the model that Stack Overflow and similar sites use as well. I know from conversation with workshop participants than only a handful (certainly less than 10%) ever post a question on SO or similar sites. I used to think the reason was that people don't want to look stupid, but I now believe that the cost of translating "this doesn't work" into a complete enough description for someone else to work with, and the number of long-latency round trips required to converge on the actual problem and its diagnosis, are contributing factors as well.

But what if we used 21st Century technology for Q&A instead of souped-up 1970s-vintage bulletin boards? What if there was a "help me now" button on Stack Overflow that was enabled whenever the person who'd asked the question was online? When clicked, it would connect the clicker and the person needing help via VoIP and desktop sharing (with some kind of "are you sure?" handshake at the outset) so that people could get real-time interactive help when they needed it. Based on what I've seen in the last five months, this would be a lot more effective than anything with the translation bottleneck and latency of a bulletin board.

This is where "counting from five" comes in. Somebody must have done this before, or done something similar enough to tell me what potholes there are on this road. Enter Scott Gray, who's been building, using, and running online educational systems for two decades, and is now the head of the O'Reilly School of Technology. We chatted today, and here's what he said:

All right: if someone with (literally) ten times as much experience as me tell me I'm off track, I ought to listen carefully and think a little more, even if it means discarding what seemed to me to be a very promising idea. Another alternative is to try to find a really cheap way to cobble together some low-fidelity experiments to get a better feeling for how this would work in practice—the equivalent of a napkin sketch in UI design. If you have insights to share, please share 'em.

< OlderNewer >

This post originally appeared in the Software Carpentry blog.