, the Two Sigmas, and Customer Support

I had another good conversation with Mike Hoye this morning, during which he pointed out that we’re living in a golden age of independent game development. The reason, he believes, is that there’s no longer a minimum viable size for a game: one person can create and deploy a simple gem like or 2048, which gives radically new ideas a chance to become real.

Is teaching about to experience a similar Cambrian Explosion, and if so, will it be good for learners? I think the answer to the first is “yes”. More online learning has taken place in the last six months than in the whole of human history prior to COVID. Literally millions of teachers have suddenly had to figure out how to do it without the months and money it takes to produce a MOOC. They have had to find ways to do things fast and cheap, and some of their crazy ideas are bound to be better than anything we’ve done before.

To answer the second question we need to know how much better teaching and learning could be. A partial answer comes from work done by Benjamin Bloom and others in the 1980s:

…the average student tutored one-to-one using mastery learning techniques performed two standard deviations better than students who learn via conventional instructional methods—that is, “the average tutored student was above 98% of the students in the control class”. Additionally, the variation of the students’ achievement changed: “about 90% of the tutored students…attained the level…reached by only the highest 20%” of the control class.

Fifteen years ago, the MOOC model of recorded video and autograded exercises was sometimes touted as a way to scale the benefits of one-to-one tutoring, but that has proven to be an evolutionary dead end. What intrigues me now is the way good customer support teams teach every day:

  • Customer: “I have a problem.”

  • Support: “Let me ask a couple of questions. Hm. OK, it’s not a purely factual matter like a missing license key. Instead, it looks like I’m going to have to explain X in order for you to understand how to solve it.”

Every customer has a different background, so X varies from call to call, but some X’s come up so often that support staff get a chance to figure out how to explain them well. The challenge is how to transfer those good explanations to their colleagues. “Write it down” is the obvious answer, but:

  1. There are half a dozen other customers with issues that need to be solved right now. And yes, a little time now might save other people a lot of time in the future, but only if this topic actually does come up frequently.

  2. The details of the explanation vary significantly from one telling to the next based on the teller’s impressions of their listener.

  3. “Wikis are where knowledge goes to die.” Unless someone works continuously to organize the collected explanations, what accumulates will be unfindable, contradictory, or out of date. Unfortunately, the “someones” who can do best are the experienced support staff whose phones are metaphorically ringing off the hook (see item #1).

There are echoes here of Reusability Paradox, which states that, “The pedagogical effectiveness of a learning object and its potential for reuse are completely at odds with one another”. It also reminds me of a long-ago colleague’s explanation of why he only wrote software documentation on demand: he could never tell in advance which parts people would actually have questions about.

I don’t have any conclusions yet, much less any advice, but I predict that we’re going to see a lot of teachers shifting to a “you call me any time” model with late-teen and adult learners, and that support for the choral explanations exemplified by sites like Stack Overflow and Quora are going to become much more common. What I need to find now is a good introduction to any research that has looked at the on-demand teaching done by customer support teams and how members of those teams share their greatest hits with each other. If you know of any, I’d be grateful for pointers.