The Wes Mongtomery of Software

I had a conversation yesterday with a data scientist who is now working for a large IT consulting company. We got to talking about the tension between the scripts that data scientists cobble together to get a particular answer and the multi-level cloud architectures with complete integration test suites that software engineers believe are the “right” way to build software that routinely makes hundred million dollar decisions. In my mind the argument about “hacked-up scripts” versus “over-engineered collections of microservices” arises from a conflation of two distinct ideas; since quadrant diagrams are almost always a sign that someone’s trying to sell you bullshit, here’s mine:

Four quadrants of coding

The Beatles’ version of “Twist and Shout” is harmonically, melodically, and rhythmically much simpler than Coltrane’s 1967 recording of “My Favorite Things”, but both tracks were recorded in a single take. In contrast, Ravel spent months rewriting his “Pavane for a Deceased Princess” and Beethoven spent almost two years tearing his Ninth Symphony apart and putting it back together. While every single note of each was carefully considered, the former is once again much simpler than the latter.

But here’s the thing: Coltrane and the Beatles were able to create something great in a single take because they didn’t actually do it in a single take: they had both played those particular songs many (many) times before doing it in the studio. Similarly, if you watch Bruce Springsteen and the E-Street Band play You Never Can Tell for the first time, they pull it off because they’ve played songs like it literally thousands of times.

I think the same can be true of software. I can’t count the number of tools I’ve written over the last forty years to transform text from one format to another; asking me to write another is like asking a seasoned session musician to play a twelve-bar blues, or asking the data scientist I was speaking to yesterday to fit a linear model to some time-series data. We don’t need to write design docs or unit tests this time because we’ve worked through the problem and its variations so many times before that our fingers are just going to find the right chords.

Expertise doesn’t emerge without reflective practice: if you want to be Wes Montgomery, at some point you have to lock yourself in your room and play standards for a year. I wish more people who program and do data science knew that; more, I wish the people who hire them knew it too and gave them the time to practice so that they could do something great in just one take.

Shake it up baby…