Proficiency

Edwin Thoen recently tweeted a short video showing how to use multiple cursors in the RStudio IDE. It’s a powerful little feature that most users don’t know about, and seeing it reminded me of a conversation earlier this week with Andreas Stefik about proficiency and productivity. Here’s the study I’d like to see:

  1. Round up a bunch of students who use a particular IDE: Emacs, VS Code, RStudio, whatever.
  2. Count how many of their chosen tool's features they know how to use without checking reference material.
  3. Time how long it takes them to correctly solve a set of representative one-page programming challenges, where "representative" includes debugging and refactoring as well as adding new features.

What correlation do you expect to see between tool proficiency (measure #2) and productivity (measure #3)? I predict there will be some, but it won’t be a straight line: many good programmers stop exploring their IDE’s capabilities once they know “search and replace” and maybe “comment/uncomment region”. They still do more complex tasks, but they use brute force (producing the “pocketa pocketa pocketa” sound one occasionally hears keyboards making in coffee shops).

Now here’s the second part of the study:

  1. Show those students how to use half a dozen features of the IDE that they're not currently using.
  2. Require them to master those features by giving them a lab exam (described below).
  3. Repeat #3 above to see if they are now more productive.

The lab exam I have in mind is simple: turn on screen recording, do a set of tasks, then submit the video showing how you did them. Based on a very small sample years ago, I believe you can watch these videos at 5X speed and still get an accurate impression of whether subjects are using the tool as intended, so if teaching people how to use tools well and requiring them to demonstrate that they’ve learned to actually does make them more productive, this study could feasibly be turned into a routine classroom exercise.

The deeper idea here is that we’re pretty good in computing at assessing what students create (the nouns), but we suck when it comes to assessing how they create it (the verbs). I had to do lab exams in chemistry and electrical engineering because my instructors wanted to be sure that I could use the equipment well. It was partly a safety concern (you don’t really appreciate your eyebrows ‘til they’re gone), but they also wanted us to be able to focus on what we were trying to accomplish and not be distracted by the mechanics of accomplishing it.

Watching Thoen’s video makes me wonder how many seconds programmers lose every minute doing things that their tools could do for them, and how many minutes they lose every hour because doing those things by hand distracts them from whatever they’re really trying to accomplish. As always, if you’re a grad student with time on your hands, I’m easy to reach and would enjoy chatting…

Updated: