Benchmarking Languages


Back in the 1980s, R.W. Hockney introduced two measures for quantifying the performance of pipelined machines. The first, r, is the pipeline’s maximum possible performance on an infinitely large data set. The second, n½, is how much data you need in order to reach half that theoretical peak, i.e., how big your problem has to be to offset startup overheads. In the mid-90s, I suggested that another performance measure would be equally interesting: how long it takes to write code that achieves half a machine’s rated performance. I called it p½, and observed that for many supercomputers it was effectively infinity, since no real application ever achieves more than 10-20% of the performance that manufacturers claim.

I no longer believe that measuring time-to-solution is the right approach, but I still think that we can learn a lot from comparing implementations of common problems in different languages. The Cowichan Problems that I proposed for measuring p½ aren’t right for comparing the general-purpose languages vying to be the next Python, but I think we’d all learn a lot if people implemented a few of the examples from Software Design by Example in Rust, Zig, Nim, Haskell, Elixir, F#, Scala, and what-not. My choices would be:

Between them, these five examples touch on most of the core design ideas I used when programming, such as actions as objects (the editor), hashing (the file backup tool), dependency management (the build manager), introspection (the style checker), and asynchronous I/O (the web server). They are also a nearly-complete minimal development stack—only “nearly” because I felt the ideas in a unit testing framework overlapped enough with those in a style checker that the set only needed one or the other.

I think these problems are different enough from each other that each one’s implementation would highlight something about the language that would otherwise not be seen. If you’re teaching a software design class with an interesting new language and want to give these to your students as assignments, please let me know how it goes.