A Dynamic Languages Symposium will be held in Montreal on October 22 (in conjunction with OOPSLA). I’ve put my interest in extensible programming on hold to pursue other projects, but I’d still like to attend. Anyone else planning on going?
I’ve been saying for a while now that extensible programming systems are the Next Big Thing, but I wasn’t expecting them to arrive this quickly. Check out Expressive Programs Through Presentation Extension, by Eisenberg and Kiczales (at UBC). They use Java plus annotations for storage, rather than XML, which allows them to piggyback on the standard Java/Ant/JUnit/Eclipse toolset. They even provide (simple) graphical embeddings; the only thing missing from the core I asked for in 2004-05 is debugger support. Anyone wanna do a PhD? 
AntUnit 1.0 has been released; it allows programmers to define test cases inside an Ant build file, and proves that every build system (or anything else with a configuration file) turns into a programming language.
One of the projects I’m contributing to these days is writing a first-year Computer Science textbook using Python. We’re using DrProject to manage it: after all, LaTeX files are really just another kind of source code, and what better way to keep track of who’s supposed to be doing what than ticketing?
Well, since you asked… The truth is, we’re storing to-do information in two ways: as tickets in DrProject, and as specially-formatted text in the LaTeX. All the big items use the former; the little notes to ourselves like, “This sentence is cheesy,” are inside the .tex files like this:
\FIXME{This sentence is cheesy.}
Embedding “tickets” in the source is a bad idea for several reasons. First, the embedded items are invisible to DrProject: they can’t be searched, ordered, assigned to particular users, and so on [1]. Second, whenever you store information in two places, you run the risk of duplication, contradiction, or omission. Right now, we have no way of knowing which of our FIXMEs are also recorded as tickets, and which aren’t; we could ask people to file a ticket each time they create a FIXME, and delete the FIXME when they close the ticket, but that’s a lot of extra work.
Despite these problems, embedding little notes in code is such a popular working practice that Eclipse and other IDEs have tools to collate and present markers of this kind. The reason is simple: embedding in code is easy. Even if you have Mylar installed, so that you can work with your ticketing system from within Eclipse [2], throwing a TODO comment or a FIXME macro into your source file disrupts your train of thought—your flow—much less than filing a ticket [3].
There’s another, subtler issue here as well. Suppose you did want to file a ticket to say that a particular sentence was cheesy, or that a particularly complex assignment statement should be refactored. Would you quote it in the ticket? There goes your flow, but what else can you do? You can’t point to it (e.g., quote file name and line number), because the text or code in question might be reorganized between the time the ticket is created, and the time someone gets to it. The only thing to do is to follow where literate programming led, and Javadoc half-heartedly followed: store the “documentation” with the “code”, and tell ourselves that it’s the least of the available evils.
But wait: our source files are under version control, aren’t they? And DrProject can see the version control repository. Why can’t DrProject scan the files in the repository, extract the FIXMEs and TODOs, and turn them into tickets? Better yet, why not have it look for FIXMEs like the one above and insert an automatically-generated ticket ID, i.e., turn the FIXME into something like this:
\FIXME[179]{This sentence is cheesy.}
Those “tickets” can then be managed like any others: if someone closes one in the database, DrProject can delete the corresponding line from the source file, and vice versa.
What I’m really proposing is to treat information-in-the-database and information-in-the-repository on an equal footing. At some level of abstraction (which we have to define and implement), it shouldn’t matter how or where the ticket is stored. All that really matters is the information it contains, and the operations users can perform on it. If it’s easiest for them to enter that data by adding a line to their source code, great—we can handle that. If there’s enough data to justify them switching tools (e.g., a one-page description of how to reproduce a complicated synchronization bug), we can support that too.
It’s tempting—but it won’t work. The problem is that the editors people use when they’re working with source code are unstructured. The editor I’m using to create this posting knows about HTML; if I press the < key on my keyboard, it adds the string < to the file. In contrast, the editor in Eclipse lets me put whatever I want in my Java files---even text that can't possibly be legal Java. We would therefore have to trust users (a) to format their FIXMEs and TODOs exactly the right way when initially adding them to files, and (b) not to mess up any of the information the system added. Experience with first-generation CASE tools and similar systems proves (at least to me) that people will get both wrong often enough to find the system a hindrance rather than a help.
Teaching Eclipse's editor how to format a \FIXME[...]{...} in a .tex file is not the right answer: different issue tracking tools will have different conventions, and anyway, what do we do when someone wants to add \CODEREVIEW{...} or \QUESTION{...} or something else? The right answer is to allow developers to create custom micro-editors and bind them to particular flavors of micro-content. The document then becomes an assemblage of strongly-typed elements, the presence of which causes display/modify/diff/merge handlers [4] to be loaded and run.
So once again it comes back to extensible programming. We separate models, views, and controllers when we're building tools for other people, but we still, in the early 21st Century, insist that our files be unstructured text. It's easy to see the cost of changing---legacy tools would stop working, and we might have to (shock horror) put Vi or Emacs out to pasture. I think it's time we started thinking about the cost of staying stuck in the 1970s; I think we ought to start paying attention to all the neat tools that aren't feasible to build because we're afraid of embracing the very future that we've dedicated ourselves to creating.
\FIXME macro to record a user ID, but then we'd have to validate it. And what about priority? Should \FIXME have as many parameters as tickets have fields? Brr...
[2] DrProject doesn't support Mylar yet, though there is a plugin for Trac. If anyone is looking for a challenging, useful CSC49X project...
[3] The problem isn't the time it takes to fix whatever you've noticed; the problem is that you have to put aside whatever problem you were thinking about to do so. People talk about "pushing" and "popping" issues on a mental stack, but the brain doesn't actually work that way: lots of studies have shown that it takes several minutes to get back in a flow state after any significant interruption.
[4] Display and modify should be obvious; diff and merge are needed so as not to discourage users from putting files containing such content under version control. (I would give you real cash money right now for an Excel merge-and-diff tool, and no, export as CSV and use text diff is not an answer.)
A couple of years ago, I wrote an article about extensible programming systems, which I believe are the Next Big Thing in programming. I’ve had a few pings about this recently, so I thought I’d post my current link collection:
If you know of others, I’d welcome pointers.
This article, by Eamonn McManus, is a nice little summary of API design principles. It contains a bit of motherhood and apple pie—nobody would ever set out to make an API difficult to learn or hard to use, for example—but the specifics are good (particularly the discussion of why interfaces are often the wrong thing to use). The article contains a link to this essay at the NetBeans site, which talks about some of the same ideas in more detail.
Together, the articles got me thinking: why don’t we ever talk about or document a module’s external programming interface (which I hereby dub “XPI”)? This is the classes, methods, and system calls that the module depends on; more particularly, it is their semantics. Design by contract allows a piece of code to specify what it provides; allowing that code to specify what it requires using similar pre- and post-conditions would be a big help in managing the asynchronous evolution of libraries that makes postmodern programming so hard.
Following links from the latest Subtext demo, I came to Martin Fowler’s article “Language Workbenches: The Killer-App for Domain Specific Languages?”. It’s well written and thought provoking, like everything else Martin writes, but I think there’s one glaring oversight. Two thirds of the way through, he says:
…there are three main parts to defining a new DSL [Domain Specific Language]:
- Define the abstract syntax, that is the schema of the abstract representation.
- Define an editor to let people manipulate the abstract representation through a project.
- Define a generator. This describes how to translate the abstract representation into an executable representation. In practice the generator defines the semantics of the DSL.
That’s great — but #4 should have been:
- Create an inspector that allows people to debug their DSL’s behavior in its own terms.
Now, you could claim that the debugger was implied by #2 (the editor), but i practice, it never works that way. Time and again, I see people struggling with systems that don’t provide tools for debugging: Makefiles, Antfiles, config.xml files for servlet containers, and on and on and on. When something goes wrong, the only options are (a) tweak it until it converges on what you wanted, or (b) debug the implementation of the nice abstraction you were just thinking in.
I think that the inability of today’s C++ debuggers to display something useful when confronted with templated code is the singlest biggest obstacle to their wider use. I think that Java made a horrible mistake in deciding to erase type information when compiling generics, so that what’s actually executing can only say, “Yup, it’s another Object.” Extensible languages could revolutionize software development, but it’ll only happen if we focus on closing the debugging gap.
So, anyone looking for a Ph.D. topic?
There’s a new demo of Jonathan Edwards’ Subtext system online. This is the best example I’ve seen yet of an extensible programming system. It’s pretty cool—once you stop insisting that programs have to be directly presentable as ASCII strings, many new thoughts become possible.
Quote: “It’s good to remember that with practice, you can even learn to read Perl. Humans are not good at abstract thinking, but they’re great at pattern recognition. I think it’s a win to trade one for the other.”
(And to follow up on the “many new thoughts become possible” meme, I once saw a talk by Richard Dawkins on “The Evolution of Evolvability”. Dawkins’ thesis was that some evolutionary changes give evolution new things to act on—segmented bodies, for example, allow for specialization of segments and limbs. Code-in-database systems almost certainly have this property.)
My ACM Queue article on extensible programming systems just got slashdotted. Once again, it’s clear that most of the posters haven’t bothered to read the article: even the headliner seems to think that I believe programmers will all be typing XML tags five years from now.
The article’s real point is that the next revolution in programming will not come from aspect-oriented languages or new ways of expressing concurrency; it will come from extensible languages—syntactically and semantically extensible, just like Common Lisp and Scheme. These languages will require us to turn programming tools into extensible frameworks, which will in turn finally force us to adopt the model/view separation that we’ve been telling the rest of the world to use for the last twenty years. For historical and marketing reasons, those models will probably be stored as XML, but programmers won’t look at the tags any more often than they look at assembly code.
Lots of people are already working on systems of this kind; just this week, for example, I came across the Proxima editor, a generic presentation-oriented editor for structured documents which does many of the things I think the next generation of general-purpose editors will have to do. Who knows? Maybe the Slashdot article will turn up a few more like this… And maybe Toronto will win the Stanley Cup this year.
Postscript: I was very pleased to see Jon Udell include a mention of the article in his blog. Some of the links from his post look very interesting; looks like I’ve got some more reading to to…
http://www.cs.umd.edu/~pugh/IsCodeOptimizationRelevant.pdf
Recent Comments