The Third Bit

These links accumulated while I was in Peru:

• Why I Hate Frameworks, from the Joel on Software discussion group. Funny, but am I the only one who found the casual reference to killing ex-girlfriends with hammers disturbing? The proportion of women going into most CS programs is still declining, and as Margolis and Fisher showed, violent and misogynist language is part of the reason. This is why I no longer review books based on “programming as war” metaphors, like Faroult and Robson’s new The Art of SQL. (Coincidentally, I read Danc’s piece on shock game advertising just a few minutes after the Spolsky piece…)
• This roundup of consensus web filters opens with a clear description of what these Web 2.0 tools are trying to do, and how best to use them. Kevin Kelly’s site also showcases Cuboro, which he claims is the best toy since Lego.
• Chris Bliss juggles to the music of The Beatles. Of course, calling what he does “juggling” is like looking at a Picasso and saying, “Ah, paint.”
• A long and thoughtful post from Michelle Levesque about Google’s decision to censor content in China. Michelle’s about to go work for Google; I feel a little better about their future plans knowing she’s going to be there. “We’d rather have the bad guys running everything // Than have nobody running things at all.”
• This retraction and deletion, from science journalist Carl Zimmer, is a perfect example of responsible blogging. As he says, deleting posts you have discovered to be in error isn’t a perfect solution, but it’s the best available for now.
• Confessions of a Virtual Intelligence Analyst (plus the outbound links) is a fascinating look at on-line surveillance in gaming worlds. “Guild churn”, anyone?
• Eric Sink predicts the demise of developer-oriented magazines. Sadly, I agree; guess I should start looking for another (for-pay) outlet for book reviews… On the other hand, if Lee Falin’s article is right, maybe there won’t be any in a couple of years.
• A screencast of Spore, which might turn out to be the Best Game Ever [tm].
• And finally, Math for Programmers, from Steve Yegge (via Harald Koch). There are some interesting ideas here for educators (and everyone else, too).

Uncategorized

Update: Previously, I discussed some of the issues we encountered by switching to Kid for our templating framework. While Kid was much nicer to work with, we were surprised at how slow it was. Chris Lenz, however, corrected me by pointing out that inserting XML fragments into the template (via the XML(...) function) should be avoided whenever possible. Instead, your controllers should be creating ElementTree objects that can be rendered directly by Kid.

For example, instead of passing the string <em>foobar</em> to your template, pass an ElementTree:

from elementtree import ElementTree element = ElementTree.Element('em') element.text = 'foobar'

In addition to the substantial performance boost, using ElementTrees for embedded HTML adds an extra level of security. Because most templating frameworks just stick HTML strings into a template, the developer must always ensure that those fragments are properly escaped. Fortunately, this is done automatically when you use ElementTree/Kid.

With this in mind, we set out to replace hard-coded HTML fragments with ElementTree objects. We soon ran into problems, however, when we realized that nearly all of our embedded HTML was generated by our custom WikiParser engine and third-party syntax highlighting tools. Refactoring the existing WikiParser engine might qualify as one of Dante’s Seven Stages of Hell, and I don’t think we could reasonably expect the authors of SilverCity and Enscript to use elementtree

For the time being, we decided to remove syntax highlighting from DrProject. While this is a nice feature to have (see here for an example), incorporating these tools with Kid was just too difficult. The HTML returned by enscript is not valid XML, so you cannot embed the output into Kid nor can you parse the output to create an ElementTree without a pre-processing stage. In the future we plan to revisit this issue, as the “ooo” factor for syntax highlighting is a big plus!

Furthermore, we decided to rewrite the WikiParser from scratch, using a proper parsing engine. Liam Stewart wrote the parser using Plex. We have added a custom formatter to build the ElementTree representing the parse tree. The resulting object can then be passed directly to Kid, without further processing. The new WikiParser is much cleaner, and much easier to maintain (at least, we think so . The only drawback we have encountered so far is speed; the new parser is quite a bit slower than the old engine. The performance boost we have gained by not having to perform XML(...) calls in our Kid templates is nearly offset by the time it takes to produce the ElementTree!

While we are currently trying to optimize the parser as much as possible, this performance hit may be unavoidable with a pure Python parsing library. If you have experience with other Python parsing libraries, let us know. One caveat: we would prefer to keep the library 100% Python; while using a C-based parsing engine would be much faster, we also have to think about portability and maintainability. DrProject already has enough dependencies as it is; we don’t want to add a whole set of C related libraries.

DrProject

This article, by Sun’s Bart Smaalders, describes ten common performance mistakes software developers make. One example:

During Solaris 10 development, Solaris engineers fixed a long list of performance problems across the kernel and user libraries. Toward the end of the release, [they] spent some time reviewing just what had been improved and by how much—and what was the underlying cause of the performance problem. Interestingly enough, all the really big improvements (above, say, 200 percent) resulted from changes in algorithms. Over and over again, all the other performance fixes—using specialized SIMD processor instructions such as SSE2 or VIS, inserting memory prefetch instructions, cycle shaving—paled in significance compared with simply going back and rethinking the locking algorithms and/or data structures.

Uncategorized

The official Python web site finally has a new 21st Century look. Guess that means I’ll have to update Third Bit some time soon

Uncategorized

This article, via Joey deVilla (again), explains the “new new rules”:

1. More features isn’t better, it’s worse.
2. You can’t make things easier by adding to them.
3. Confusion is the ultimate deal-breaker.
4. Style matters.
5. Only features that provide a good user experience will be used.
6. Any feature that requires learning will only be adopted by a small fraction of users.
7. Unused features are not only useless, they can slow you down and diminish ease of use.
8. Users do not want to think about technology: what really counts is what it does for them.
9. Forget about out the killer feature. Welcome to the age of the killer user-experience.
10. Less is difficult, that’s why less is more.

They all make sense, but I wonder whether they’ll last any longer than the old “new rules”…

Uncategorized

The lecture on server-side programming is up: if you think it should be split into two parts, please let me know. That leaves only three to go (XML, security, and integration).

And by this time on Thursday, we’ll be in Lima, Peru…

Software Carpentry

This diagram from Joey de Villa is the best explanation I’ve seen yet of the differences between Web 1.0 and 2.0:

Web 1.0	Web 2.0

Uncategorized

I just discovered the Lost Garden site, via this article on a notation for describing game play. Worth adding to your blog roll; worth asking if there’s a similar notation waiting to be invented to describe user interactions with standard desktop GUIs.

Uncategorized

You don’t often get to glimpse the future, and when you do, you usually don’t realize what you’ve seen until it’s too late to take advantage of it. Take the World Wide Web, for example: I saw a demo of an early version of the Mosaic browser in 1993, and thought, “Eh.” A few years later, Jon Udell explained blogs to me, and I had the same reaction. And Java? I still have a copy of an email in which I patiently explained to a friend why it would never make any headway against C++.

That’s why I read these two books cover to cover: they both show what part of the future of practical computing is going to look like. Oakley’s Monad describes the eponymous tool from Microsoft (also known as MSH). At first glance, it looks like just another command-line shell, but then you see scripts like this:

D:Scripts> filter ExtractUrl {$parts = $_.split(" "); if ($parts[10] -eq 200){$parts[4]}} D:Scripts> get-content ex010101.log | ExtractUrl | select-object -Unique

and you realize that something very different is going on. Where classic Unix utilities communicate via streams of strings, their Monad equivalents pass objects back and forth. Those objects can be as simple as lists, or as complex as trees and tables. Whatever they are, Monad offers simple, uniform ways to dissect, transform, and reassemble them. The result is something that’s as convenient as Awk, but as powerful as a full-blown programming language.

If anything, Oakley’s book underplays just how powerful Monad might turn out to be. Instead, he concentrates on the nuts and bolts of installing it, combining built-in filters, creating your own, and so on. The writing is classic O’Reilly: clear, concise, and technical without being intimidating. The examples are right where they should be, and the index pointed me right at the answer to every question I had. Its only weaknesses are a near-total lack of diagrams, and the fact that Monad itself uses Perlish syntax (which isn’t Oakley’s fault, but is still both a sin and a shame).

Greasemonkey Hacks is just as important, though it’s a rougher read, both because it’s a compilation, and because its authors are so fascinated with their cool new toy that they make few concessions to non-cognoscenti. Greasemonkey is a Firefox extension that processes HTML pages after the browser has received them from the server, but before they’re displayed. You describe what you want to do by writing a snippet of Javascript (where “snippet” sometimes means “ten pages or more”).

So, do you want to resize text entry fields? Highlight search terms? Create a keyboard shortcut to will zoom images? Strip out ads? This book describes user scripts to do all of these things, and dozens more. “Describes” is the operative word, though—not “explains”. I’m all for showing people code, but to take the most egregious example, do the five pages of hex codes in “Hack 89: Syndicate Encrypted Content” really add to anyone’s understanding of anything?

While Greasemonkey may seem like nerd candy, it’s actually an important step toward a user-centric web. Prior to Greasemonkey, the only control non-programmers had over how they consumed content was whether they got it by email, in their blog reader, or by jumping to something in their favorites list every once in a while. Greasemonkey changes that: most people will never write scripts, but they will share and tweak them like music and photos. By the time today’s twenty-year-olds are thirty, they’ll be as bewildered by a younger generation’s casual remixing of content as I am by my niece’s text messaging shorthand. You heard it here first, friends; you heard it here first.

---

Andy Oakley: Monad. O’Reilly, 2005, 0596100094, 206 pages, $34.95.

Mark Pilgrim: Greasemonkey Hacks. O’Reilly, 2005, 0596101651, 495 pages, $24.95.

Books

We’d like to add some simple time-series charts to DrProject. We’ve played with matplotlib (server-side Python, a pain to install) and XML/SWF Flash charts (can’t handle irregularly-spaced data, not open source, requires Flash); and I’m wondering: are there any libraries out there to do this in Javascript? Our needs are very, very simple… Pointers to me, please.

DrProject