The Third Bit

You’re never done designing a useful system, because if it is actually useful, people are always coming up with new requirements for it. For example, I’ve been using DrProject to manage my software engineering course this term: the class notes are all in the wiki (so that students can fix ‘em up), I communicate with students through the DrP-managed mailing list, and so on. But now I want to archive that material in the portal used by the departmental lecturers, so that future instructors can find it after the current portal is shut down. Moving the SVN repository over is trivial, and dumping and restoring the relevant portions of the database needs just a few lines of SQL…

…and might well result in disaster, since keys that are unique in one database might well not be unique in another. The “All” project in one portal is definitely not the same as the “All” project in another; I could replace uses of “All” as a key in appropriate places, but what about the user ID “nick”?  The odds are good that it’s the same person in both portals (since they’re both hosted by our department, and we use unique Unix IDs wherever possible), but it’s not guaranteed.

There are subtler problems too. Suppose there are no conflict in the user IDs I need to import to make ticket ownership and wiki page authorship references resolve.  I go ahead and add the users, import the data, and now the database believes that “nick” created some tickets and sent some email messages before his account existed.

How important is this?  Not very—there’s no real cost to leaving the old portal up.  How irritating is it?  Quite—I like to keep things tidy, and having all the old courses in one place just seems like the right thing to do.  (We also had some people from California approach us earlier this week about converting their existing Trac instance to DrProject, during which a similar problem would arise.)  Never done…

DrProject

Google announced this year’s Summer of Code projects today. I’m very pleased that six students from the University of Toronto were among the recipients:

• Matthew Basset / Hackystat: Hackystat Sensor for Visual Studio Team Foundation Server
• Eran Henig / Python: Python Plugin for Web-CAT
• Victoria Mui / GenMAPP: Automatic (Smart) Node and Edge Label Layout
• Wong I Wa / Hackystat: Visualizing HackyStat Data Using Flare
• Qi Yang / Eclipse: Web-CAT Grader Plugin for Eclipse
• Qiyu Zhu / Python: DrProject Web Administration Interface

Four students from other schools (Jeff Balogh, Luke Paireepinart, Daniel Servos, and Artem Yunusov) will be doing projects we’re involved in, and of course there are our non-GSoC students:

• Nicole Allard and Dmitri Vassiliev will continue their work on the feature diagram plugin for Eclipse and browser plugins for SlashID respectively.
• Geofrey Flores and Kosta Zabashta have been awarded an NSERC USRA to study the usability and productivity of web-based and IDE-based grading tools (GF) and to integrate instant messaging into DrProject (KZ).
• Luke Petrolekas will be testing and documenting DrProject (and probably other projects as well).
• Nick Jamil will be doing miscellaneous DrProject development.
• Chris Maddison and Hardeep Singh are going to help beef up the Python multimedia libraries we use in first-year teaching thanks to an ITCDF grant.

Oh, and we mustn’t forget the graduate students:

• Samira Abdi is using machine learning techniques to cluster events in project histories to make them easier to visualize.
• Jeremy Handcock is building tools to visualize software project histories (clustered and otherwise).
• Carolyn MacLeod will be studying how novices use modeling tools like SPIN and Alloy to see if there are systematic patterns in the mistakes they make.

Me? I’m getting married (twice), helping Jennifer, Paul, and Jason finish off a CS1-in-Python textbook, and trying to find some funding. It promises to be a busy sixteen weeks…

DrProject, Teaching

A year and a bit ago, I posted pictures of DrProject‘s database schema and architecture. I’m happy with the former — the only thing I’d do differently is make it larger and add comments (or change field names to be more descriptive) — but the latter still bugs me.  It doesn’t carry as much information as a napkin sketch of a user interface, much less an architectural blueprint.  I’ve looked at lots of books on software architecture (Reekie & McAdam and Gorton being my favorites), but I still haven’t found a notation that seems like good value for time.  If you know of one, I’d welcome a pointer…

DrProject, Teaching

We’ve been having a problem recently with self-registration in the new version of DrProject. Would-be users fill an oh-so-familiar form (preferred ID, email address, password); their data is then held in queue for an admin to approve. However, when the admin clicks “approve”, DrProject reports “user already exists in password file”.

Yesterday, David Wolever managed to track it down.

1. Two users are being confirmed at once (i.e., there are two or more requests pending, the admin has selected “approve” for both, then clicks “submit” on the form).
2. One works fine.
3. The other triggers an exception for some reason (usually missing information).
4. The exception causes the database transaction to roll back (good), but the first user’s ID and password are in the external password file (bad).

Yes, we will improve pre-transaction validation so that #3 happens less frequently, but we’re still left with the basic problem: we can’t make operations across two things (in this case, the database and the file system) atomic. We could make up a list of file operations to be undone in case of a database transaction failure (i.e., roll our own transaction system—bleah), or do the file operations first and proceed to the database transaction only if the file op succeeds (more code, which means more places where developers could forget to do things), or move passwords for self-registered users into the database (which makes administration of large portals harder: managing credentials in multiple credential stores is a project unto itself).

I’m sure we’ll come up with something, but until we do, I’m just going to savor the irony of it all. Four years ago, when we forked from Trac, we faced a similar problem. Should each of a portal’s projects have its own database and version control repository, or should we use one DB for all projects (but separate repositories), or one DB and one repository? We eventually realized that one DB shared by all projects was the only option that made sense, even though it meant more hacking on the tables inherited from Trac, because multiple DBs would require us to build our own atomicity layer. It’s ironic that trying to keep all the user credentials in one place (a file on disk, accessed only by a setuid validation program), has gotten us back to the same point.

Later: at Alan Rosenthal’s suggestion, David Wolever has “solved” this particular problem by making operations on the credentials file idempotent: if DrProject tries to add data that’s already there, nothing happens and no error is reported.  It’s not a general solution—in fact, I’m sure that one day someone somewhere will curse us for special-casing this—but it’s good enough to get 3.0.1 out the door.

DrProject

I skipped an important step in my previous post: I wasn’t explicit about the features something had to have to qualify (in my mind, if no one else’s) as a “software project portal”. Here are my current thoughts, sorted in order of importance.

1. Identity management and access control (i.e., accounts, privileges, and all that jazz).
   • Bonus marks if privileges are organized into roles for easier administration.
   • More bonus marks if administrators can define new roles.
   • Still more if it integrates with OpenID or the like.
2. Issue tracking (a.k.a. “ticketing”), or some other kind of to-do list.
   • …with a query interface that lets users filter and sort.
     • Bonus marks if F&S settings can be saved and shared, so that users can set up things like “all the high-priority items assigned to me that are due within a week”.
   • More bonus marks if fields in tickets can be customized without rewriting any code.
   • Still more bonus marks if this is combined with a calendar so that users can group tickets by release or due date.
     • DrProject‘s “milestones” have been inherited from Trac, and are just as lame.
3. Version control repository browser (browser only because there’s no safe and easy way to do a commit through a browser).
4. Mailing list management (even though Trac doesn’t have this, and I still consider it a portal).
5. A wiki.
   • Marks taken off if the wiki’s grammar doesn’t have shortcuts for linking to tickets, code revisions, mail messages, and the like.
6. Cross-component search.
7. A plugin or extension system.
   • DrProject‘s needs a major overhaul…
8. Some kind of over-the-web API (REST, SOAP, or RPC doesn’t matter, just so long as it allows remote scripting, automation, and integration).
9. An integrated view of the project’s history.
   • This should be available as an RSS feed as well as in the browser.
   • Users should be able to get more immediate notification of selected events (e.g., email when tickets they care about are created or changed).
   • There should also be a separate, more detailed event log for auditing and administrative purposes.
   • Plus charts and other analytic tools.
     • We’re going to try again this summer to build something like this for DrProject.
10. Project blogs.
11. Continuous integration.
    • Most systems rely on external tools like CruiseControl, in which case this becomes “integrate with continuous integration”.
12. Test case management.
    • Actually, most don’t bother to do this separately (Rally being a notable exception). I’m still trying to figure out why nobody has integrated with FitNesse yet…
13. Requirements management.
    • A lot of systems, including DrProject, just use tickets for this, but Mingle and other agile-oriented portals support user story cards or the like.
    • As an aside, I think it’s pretty telling that nothing smaller than ClearCase offers, or even integrates with, a “classic” requirements management tool like DOORS.
14. Time tracking.
    • This comes near the bottom of my list because I believe most people input random numbers.
15. Integration with IRC, instant messaging, VoIP, and other communication tools.
    • Anyone?
16. Support for internationalization and localization.
17. Forums or some other kind of bulletin board system.
    • I personally think this is redundant given mailing lists and a wiki, but a lot of portals offer them, and we regularly get requests to add them to DrProject.

One more consideration is that a portal should be installable, rather than a hosted service: not every project is open source, and many universities aren’t allowed to store student information out of their own jurisdiction.

DrProject

We’re hoping to release a new version of DrProject next week, and persuade some Trac users to upgrade. (Multiple projects! Mailing lists! Role-based access control! Scripting interface!) This is therefore a good time to take a fresh look at what other systems offer:

SourceForge: not the first web-based software project portal, but certainly the best known and (probably) the most widely used; not free, and too big for most student projects and startups (though there are lots of cases of both using it).

Google Code: much smaller, but growing fast; only available as a hosted service (which rules it out for course projects in many jurisdictions, and for companies that want to keep their software behind their firewall).

Trac: probably the most popular entry-level open source system; this is what we forked DrProject from, and what we’re hoping to supercede.

Mingle: a relatively new offering from ThoughtWorks specifically aimed at agile projects (and lovers of sticky notes everywhere). Very attractive, but not open.

Rally, VersionOne, ScrumWorks, TargetProcess, and Acunote: same story as Mingle.

OpenProj: an open source alternative to Microsoft Project, available both on the desktop and as a service.

XPlanner, ExtremePlanner, ProjectCards, XPStoryStudio, PlanningPoker, and Plan B: all target agile processes, but lack some or all of the features of an all-purpose portal.

Perforce: my favorite version control system, which also has simple task management, but not the rest of the features a team needs in a portal.

ClearCase: a configuration management tool rather than a portal; definitely not something to inflict on a small team (or a large one, for that matter).

Jazz: “an IBM Rational project to build a scalable, extensible team collaboration platform for integrating work across the phases of the development lifecycle.” Slightly smaller than Greenland, and not yet finished; definitely not for student teams.

So, what have I missed?

DrProject

There was a brief flurry of email on the DrProject list this week about using JSON instead of XML for communication between clients (in our case, browsers and the occasional client-side script) and servers.  The younger members of the team were excited about the advantages: less typing, easier to read, faster to convert to/from data structures, and shiny newness.  I said “no” because of the disadvantages: every implementation works slightly differently (XML may be broken, but at least it’s broken the same way everywhere), and it introduces a serious security risk [1] — I might not be able to think of a way to exploit it if all connections are HTTPS, but I’m just one guy, and I’d rather not require DrP’s users to bet that I’m smarter than all the villains in the world.

The whole discussion was an interesting reminder of how my priorities have shifted over the last ten years. I would have been on the juniors’ side in the mid-1990s; hell, back then, I still toyed with the idea of teaching people Scheme as a first language.  I guess part of growing old as a technologist is caring more about not stepping on traps than about missed opportunities…

[1] There are two ways to process JSON: use eval(), or parse it. If you’re using eval(), you are taking the risk that someone has embedded function calls in the “data” they’re sending you, in which case you just handed them the keys.  If you’re parsing, then what’s the advantage over XML, for which there are many well-tested out-of-the-box parsers?

DrProject

Yesterday was Family Day, a new Ontario holiday.  It was also the start of Reading Week (no classes at U of T), but David, Yi Qing, Luke, and Qiyu were in anyway, beavering away to get DrProject ready for PyCon while I wrote reports.  (I miss coding…)  16 new tickets created; 19 completed (but 3 reopened); 29 changesets checked in; five sub sandwiches eaten (along with three large chocolate milk, one regular milk, and an orange juice for the old guy).  We’re starting up again in an hour and a half; there’s a ton to be done, but it’s looking better all the time.

DrProject

I was talking to my software engineering class about code reviews last week, while simultaneously thinking about how to replace the wiki parser in DrProject. Two birds one stone, I decided to review a wiki parser written in Python and keep notes on what I was thinking as I went through it. I’ve posted the result as a separate page on this web site; I’d be interested in comments and feedback.

Later: Diomidis Spinellis posted a similar think-aloud back in 2004 that’s worth reading.

DrProject, Teaching

This morning the OpenID Foundation announced that Google, IBM, Microsoft, VeriSign, and Yahoo! have joined the board. We really need to get this into DrProject — I’m definitely proposing it as a Google Summer of Code project this year. (If it’s good enough for Estonia, it’s good enough for me ).

DrProject