But now I’m programming on the web, which means I’m writing Javascript. OK, that’s the equivalent of C: full of traps for the unwary, but able to do quite a bit once you get your head around it.  What I want to know is, where’s my shell?  Where’s the tool that’s not quite as expressive (try manipulating tree-shaped data in the Bourne shell), and not as fast, but lets me do with two or three commands what would take half an hour to code and debug at the base level?

JQuery isn’t what I’m looking for (though I’m very, very grateful that it exists). JQuery isn’t a watertight layer of abstraction above Javascript: it’s leaky, in the sense that users still have to pay attention to Javascript-y things.  (“Wait, I forgot to say ‘var’…”)  CoffeeScript and other compile-to-JS languages are leaky too, but the shell is as close to being leak-free as any abstraction layer I can think of. In almost 30 years of nearly constant use, I’ve almost never had to think at the C level in order to debug a problem at the shell script level [1].

So what would a shell-like tool for in-browser programming look like? Well, I think it would look something like Microsoft PowerShell. There’d be lots of more-or-less orthogonal tools, each of which did one thing, but instead of communicating via streams of text, they’d send one another streams of objects. Those tools would probably be written in Javascript, and the shell might be as well (just as bash and its ilk are written in C), but users wouldn’t see that.  What they would see is that they could package combinations of tools into scripts that could then be used as tools in their own right. The syntax for simple operations would be much simpler than that of Javascript, at the cost of some expressive power, but that’s OK: 80/20 splits are a good thing.

And hey, it would give us a chance to choose command names that are more mnemonic than ‘ls’ and ‘mv’…

[1] And the times I did were my own fault, because one of the components in my pipe was a program I’d written myself, which didn’t fork properly, and—never mind, it’s not important.

if x:
    if y:
        z == 0
    else:
        z == 1
else:
    if y:
        z == 2
    else:
        z == 3

I can write

Actually, scrap that: what I want is to be able to embed a full-blown spreadsheet in amongst other statements so that I can write:

I want to be able to embed other things as well: bubble-and-arrow diagrams of finite state machines, before-and-after diagrams of data structures, and everything else that appears in programming textbooks. I can do this if I’m writing in Word or LibreOffice, or giving a lecture in front of a whiteboard. Why can’t I do it when I’m programming?

The standard answer is, “Because programs are text.” To which I can only reply, “No: that’s a storage format for programs—a very limited (and limiting) one.” I don’t know how LibreOffice stores documents, or how SQLite stores a database, and I don’t care. In fact, I shouldn’t care unless something goes wrong (about which more later). In every domain except programming, programmers accept that there should be a division between:

1. models (the things that make up whatever we’re working with)
2. views and controllers (how we display and interact with those things)
3. storage formats (how we store things on disk)

The only place where we don’t do this is when we’re building things for ourselves. We insist that programs be stored as text streams because that’s what Emacs, Vi, and the Unix command-line tools know how to work with, and what our compilers know how to process, but it doesn’t have to be like that. We could store programs as XML documents, blobs of JSON, serialized Python data structures, whatever your favorite database uses, or something else entirely, then teach our controllers and views (i.e., our editors), and our processing tools (compilers and interpreters) to display that however we wanted and let us manipulate it in whatever way made the most sense for the content.

I thought at the time that doing this would require us to combine three specific technologies:

1. Programming languages that allow programmers to extend their syntax.
2. Compilers, linkers, debuggers, and other tools that are frameworks for plug-ins, rather than monolithic applications.
3. Programs that are stored as XML documents, so programmers can represent and process data and meta-data uniformly.

Looking back seven years later:

1. #1 was the goal of the exercise, so long as “extending syntax” is understood to mean “with entirely new displays with associated processing rules”. Scheme’s hygienic macros and Python’s operator overloading are great, but they still require you to squeeze your thoughts into a single display form. You don’t have to do that on a whiteboard, when authoring a textbook, or when writing a plugin for Firefox (though in each case, what you’re building will be more comprehensible if it rhymes with things users already understand); why should you have to do it with programs?
2. #2 is a corollary of #1: if I’m going to add X to my language, I need a way to tell tools how to handle X. The compiler or interpreter needs to know how to translate it into primitives; the optimizer might need special rules for making it fast; the debugger needs to know how to display its operation or internals, my coverage tool probably needs something as well, and so on. This is straightforward engineering: every new “thing” should be a bundle that respects various tools’ APIs, and each tool should provide an API so that people can plug things into it.
3. I was wrong about #3, partly because I still hadn’t escaped the “storage format equals model” trap, and partly because I thought that XML processing tools like XSLT would catch on and become the basis for a new generation of pipe-and-filter tools to replace the venerable Unix toolset. What I’d say today is, “Programs are stored in some format that’s amenable to low-level inspection by people who are developing new plugin forms, in the same way that SVG diagrams can be viewed as angle-bracketed text if absolutely necessary.”

So, how has reality measured up? Well, as Bill Gates once said, “We always overestimate the change that will occur in the next two years and underestimate the change that will occur in the next ten.” There has been renewed interest in programming language design in the last few years, mostly focused around the meme of functional languages being the only way to cope with massive parallelism (they aren’t, but that’s a different post). Unfortunately, all of these languages have remained trapped in the textual tarpit: whether it’s Scala, Clojure, Haskell, or something more esoteric, their authors all take for granted that a program has to be a stream of characters that can be edited by Pico or Notepad. There are, however, a few signs of progress:

• Something—no one is quite sure what—is happening at Charles Simonyi’s company Intentional Software. This talk seems to show a working implementation of many of the ideas discussed above, focused on letting business people build things for themselves. That will fail unless someone teaches those business people the principles of computational thinking, but the tool itself shows a lot of promise.
• Mathematica’s Computable Document Format doesn’t provide the extensibility, but I still think it’s worth watching. As I said in the original article, the biggest obstacle to getting from here to there is that everything has to change at once: editors, compilers, the language, etc. Vendors who own the whole toolchain, like Wolfram (Mathematica), The MathWorks (MATLAB), and Microsoft (.NET) can do this more easily than systems with distributed ownership (the C/Linux/Vi-or-Emacs/GCC toolchain). In that light, Microsoft’s compiler-as-a-service project is very interesting as well…
• Steve Witten’s TermKit is also interesting: he’s replacing the plain text streams that classic Unix tools use to communicate with streams of JSON and a rich display. This grassroots, bottom-up approach is an interesting complement to the corporate, top-down approaches above.

But the project that has me really excited is Geoffrey French’s Larch, which leverages Python and Jython to give people a forward migration path. The tutorial videos don’t really do Larch justice, and you have to remember that it’s really a proof of concept, but I’m still very excited. If nothing else, it could finally bring software visualization into the mainstream: as several studies have showed, visualizations are more useful and powerful if people create their own, but existing IDEs and languages make this hard. By making “draw this” a natural part of programming, Larch or something like it could be the next big step forward in programming.

Debuggers make programming much less painful and much more productive, but a lot of students never pick up the habit of using one.  Personally, I think this is because teachers have never figured out how to put questions about using debuggers on mid-terms: most computer science programs don’t have an equivalent of the “lab exams” that are common in chemistry and biology, and if students are never examined on their ability to do things the right way, they never have to climb the learning curve.  But that’s just a guess, and tangential to the main point of this post.

What I really want to talk about is languages like Coffeescript, and why I won’t use them.  If I write a program in C, Java, Python, or Javascript, it is translated into instructions for some kind of machine (either real hardware, in the case of C, or a virtual machine in the case of the other three).  When that program runs, the debugger can match up the source code of the program and the instructions that are being executed, so that when I say, “Go to the next line,” the debugger can do what I ask.  These debuggers are WYSIWYG: the code I wrote (i.e., the code that’s in my mind) and the operations and data that I’m debugging, line up neatly.

But if I write a program in Coffeescript, it isn’t executed directly.  Instead, it is translated into Javascript, and then that is what’s run.  If I want to debug that Javascript, I can, but I didn’t write it: a computer program did.  Yes, the bits and pieces of that Javascript correspond to my Coffeescript, but the match is not obvious, and may not even be one-to-one.  It’s as if I wrote a contract in English in terms of Ontario law, then had to defend it in court in French under Quebec law.  Without near-expert understanding of how the one translates to the other, it’s hard or impossible for people to reverse engineer what the debugger is showing them and figure out which bits of the code they actually wrote needs to change.

I first ran into this problem in the 1980s, when the C++ compiler I was working with translated my object-oriented programs into a tangle of strangely-named C functions for me to compile and run.  After a while, I figured out that if the error was in XX_@_YY_@@_ZZ, I should look for a method called XX.YY taking a parameter of type ZZ, but the C that implemented overloaded operators was ugly enough that I never really wrapped my head around it.  Debuggers eventually appeared that could handle C++ “in source”, and I’m sure that if Coffeescript proves popular, a native debugger for it will appear as well.  ‘Til then, as much as I prefer its syntax to Javascript’s, I will (regretfully) turn away…

Not too bad so far, but what happens when someone wants to update the image?

The former feels more wiki-like, and I suspect that if sketching in web pages was as easy as sketching in Word or OpenOffice, a lot more developers would draw pictures of what’s going on in their applications.

I can see two ways forward: create drawing tools that rely on the <canvas> element that are as easy to incorporate into web pages as the Javascript WYSIWYG HTML editor I’m using right now in WordPress, or hack around the problem. Right now, it looks like the latter is winning.  For example, check out Jordi Cabot’s list of text-based UML tools: all of the entries translate text like

@startuml
Alice -> Bob: Authentication Request
Bob --> Alice: Authentication Response
@enduml

into a picture like:

just as a wiki engine takes //ohmigod!// and turns it into ohmigod!. (Example taken from PlantUML.)

Now, part of me wants to weep at the prospect of ASCII art’s continued survival. On the other hand, it’s a pretty creative way to work around the continuing backwardness of web interfaces, and if people actually start using it to explain what their code is doing and how it works, we’ll all be better off.