Better is Harder than New

I gave notice at Hewlett-Packard last Friday; I've enjoyed working with the Select Access team a lot over the past four years, and have learned a tremendous amount from them, it's time for me to focus on a few personal projects (including Hippo—I really would like to learn how it works).

As part of unwinding my commitments at HP, I'm going to resign from JSR-241, the committee that is supposed to be standardizing a new JVM-based scripting language called Groovy. I would probably have done this anyway; developers have been adding an average of one major new feature every ten days for the past three months, without any direction or discipline that I can see. It's kind of sad—there was a lot of excitement about the language back in the spring, and despite four of its developers now having book deals, I think it's going to be stillborn.

Coincidentally, David Ascher (of ActiveState) introduced me to Mark Hahn last week. Mark is the driving force behind PyCS, a new Python-inspired language. Mark was interested in the paper I wrote about extensible programming systems, and wanted me on board. We swapped some mail, and I commented on his proposal to use XML as an intermediate format. I closed my comments with:

Which leads me to ask, what is the compelling feature that will make programmers ache to switch from [name of language here] to PyCS? "A tidied up Python with some concurrency constructs" sounds like an improvement over what I'm using now, but not a particularly big one. Putting the "power tools" that Lisp (esp. Scheme) developers have enjoyed for the last quarter century into the hands of the masses—now that is compelling…

That got me thinking about the difference between being new, and being better, and why the latter is so much harder to do. According to Donald Mackenzie (a sociologist at the University of Edinburgh who provided seed money for the second book I worked on), a new technology has to be roughly three times better in order to displace an old one. "Better" can mean faster, cheaper, safer, more reliable, or (I guess) more exciting; the key point is, the difference has to be dramatic to be compelling. A 10% reduction in cost simply isn't enough for people to give up the comfort of the known.

That's why I've lost interest in Groovy and PyCS: I can't think of any way in which they're that much better than what's out there already. Groovy adds some of Ruby's features to Java (but does it badly: whitespace is sometimes significant, and my students found its inconsistencies very frustrating); PyCS adds prototypes and concurrency to Python…but so what? Is that enough to persuade people to change languages? To learn a new set of libraries? To bet months of their working lives on a new language's longevity? I doubt it.

I think these groups would do better to take a leap into the unknown, and build something genuinely different. Sure, it'll probably fail (i.e., will probably only ever have a few hundred users), but the "slightly better X" approach seems guaranteed to fail. I mean, even steamrollers like Java and .NET took four or five years to establish themselves; if you're designing something to be a better X in 2004, its chances of being better than the competition in 2009 seem pretty slim…

So why Hippo? At first glance, it's a "slightly better X", where "X" equals SourceForge. Version control, mailing lists, user management… there is (deliberately) nothing new in its feature set…

…except batch operations. Hippo is designed to handle courses, in which each student (or small group of students) does her work in a mini-project of her own. This means that instructors have to be able to create dozens or hundreds of more-or-less identical projects. To do this on SourceForge (or GForge, or any of the other clone systems), you must either click and type 'til your fingers go numb, or write a script that throws SQL at the underlying database. No-one would put up with the former; the latter requires skills that many instructors don't have, and by completely bypassing the system's data model layer, pretty much guarantees that someone, some day, will corrupt the underlying database beyond repair.

So, batch operations will be Hippo's "must have" feature. We hope it'll win hearts and minds for its usability, too, but as Joel Spolsky points out in his latest article, people will forgive a little clunkiness if you give them something they really want.

Now, back to getting HSQLDB to play nicely…