The Third Bit

From http://basieproject.org:

We have decided to wind down the Basie project and put our effort into Pinax, a collection of reusable Django components designed to simplify the construction of social networking sites. We made this decision because many of the things we want in Basie are actually common to groupware of all kinds, not just software project management tools, and because students working on the project will have a better learning experience if they are embedded in an active non-academic open source community.

Our first contribution to Pinax is a repository browser component with back-ends for Subversion and Mercurial. We are now working on a mailing list component, which we hope to release in December 2010.

DrProject and Basie were a lot of fun, but now that I’ve left U of T, they’re simply not sustainable. I’m grateful to everyone who supported the project: the University of Toronto, the Jonah Group, Idée, Rypple, Rogers Communications, First Media Group, Red Hat, TUCOWS, Yahoo!, and Google Summer of Code. Thanks even more to everyone who turned that support into working code:

• Botond Ballo (University of Toronto)
• Jeff Balogh (University of Central Florida / Mozilla)
• Liz Blankenship (University of Michigan)
• Chad Cogar (University of Toronto)
• David Cooper (University of Toronto)
• François Fournier (Université Laval)
• Heather Grant (University of Alberta)
• Russell Haering (Oregon State University)
• Eran Henig (University of Toronto)
• Jackie Huynh (University of British Columbia)
• Bill Konrad (University of Toronto)
• Maciej Kozlowski (University of Toronto)
• Derek Kwok (University of Toronto)
• Pierre Lafayette (Qualcomm)
• Alex Le (University of British Columbia)
• Phyliss Lee (University of Toronto)
• James Leung (University of Alberta)
• Ian Lienert (University of Toronto)
• Josh McCarthy (Memorial University)
• Christian Muise (University of Toronto)
• Nikita Pchelin (University of Toronto)
• John Peters (University of Alberta)
• Andrew Schurman (Simon Fraser University)
• Dan Servos (Lakehead University)
• Florian Shkurti (University of Toronto)
• Guillaume Simard (Université Laval)
• Noam Sutskever (high school, Toronto)
• Tony Targonski (University of Waterloo)
• Edward Tate (Simon Fraser University)
• Chris Van Wiemeersch (Michigan State University)
• Zuzel Vera Pacheco (University of Havana / University of Toronto)
• Jason Whyne (University of Waterloo)
• Greg Wilson (University of Toronto)
• Blake Winton (Mozilla)
• David Wolever (University of Toronto)
• Veronica Wong (University of Toronto)
• Kosta Zabashta (University of Toronto)

Basie

Guillaume Simard and François Fournier are writing Selenium tests for Basie (our Django-based replacement for Trac). Over on the Basie blog, they explain why they’re using random data in their tests, rather than testing against predefined fixtures.

Basie

I keep telling my students not to over-commit themselves. It’s a shame I don’t take my own advice . Here’s what I’ve currently got on the go:

Software Carpentry teaches basic software development skills to scientists and engineers. I have 80% of the funding I need to spend a year upgrading its content and delivery. I hope to raise the last 20% of the money in the next few weeks. If I can pull it off, the major challenges will be:

1. Learning how to create effective online course material: there’s lots of handwaving out there about wikis in the classroom, but nothing substantive about instructional design for mature learners using present-day internet technologies.
2. Assessment. We don’t know how to measure the productivity of programmers, or the productivity of scientists; trying to gauge this course’s impact on the productivity of scientific programmers will therefore be something of a challenge. (One of the reasons I left industry for academia in 2006 was to figure out how to do this, but my attempts to find research funding all failed.)
3. Mechanics. Site5 only allows one shell account per domain, which makes it difficult to open up the project’s Subversion repository to other contributors. And I’ll have to choose a format for the lecture notes: LaTeX, plain HTML, S5, one of the many wiki formats… And figure out a better way to create and manage images and video. And pick a bibliography format. And…

A professional Master’s degree in Computer Science at the University of Toronto to complement the department’s existing research Master’s. The program consists of five regular graduate courses, a course each on business skills and professional communication, and an eight-month industrial internship in which students have to show that they can translate theory into practice. We are now accepting applications for September 2010 entry, so if you’d like to learn leading-edge ideas from some of the best researchers in the world, please check it out.

Basie, our replacement for Trac, built on Django and jQuery, is coming along nicely, but I don’t know what will happen to it once I leave U of T. A few non-students are now involved in its development, but we aren’t big enough to bid for our own Google Summer of Code students. If anyone would like to get involved, please give me a shout. (I’d particularly like to hear from ex-project students—it would be nice to have an excuse to stay in touch.)

UCOSP stands for “undergraduate capstone open source projects”. Since September 2008, undergraduates from several universities in Canada and the US have been taking part in joint capstone projects in order to learn first-hand what distributed development is like. Each team has students from two or three schools, and works for a term under the supervision of a faculty or industry lead on an open source project. We’re currently trying to find $35,000 to hire a half-time administrator to run the program from September 2010 so that we can scale up from the present 45 students/term to 80, 90, or more. Again, if you’re interested, please give me a shout.

CSC302 is my regular undergraduate software engineering course. This term, six teams of students are porting Django to Python 3, adding pivot tables to Gnumeric, parallelizing parts of ILUTE, upgrading PyLint, pluginifying Selenium, and extending SpatiaLite. It could be the last regular course I teach at the University of Toronto; it has been a bit bumpy, but I’m glad the students are getting to work on real things.

Grad student supervision: Alecia, Zuzel, and Mike all have topics nailed down, and Jason is writing up. I plan to spend one morning a week in the department working with them from now through next January; I’m looking forward to seeing what they produce.

The Cowichan Problems. This one goes back to the mid-1990s, when I first realized that human performance was at least as important to overall productivity in computational science as machine performance. The idea is to use a suite of fairly simple applications, all stitched together, to benchmark the usability of parallel programming systems. A couple of undergrads updated the code last year; I’m hoping to revisit it as part of my work on Software Carpentry.

Book #1, called What Really Works?, is a Beautiful Code-style book that presents evidence-based results in software engineering. Where do bugs actually come from? Does pair programming get the job done faster? Can code metrics predict post-release fault rates? Are some programming languages intrinsically more productive than others? Each of our authors will explore one such question in a chapter-length essay; contributions are now coming in, and we’re still on track to have the book on the shelves this summer. (I’ve been talking about this subject and this book for a few months now; if you’re interested, you can view the slides.)

Book #2 is yet another collection, this time exploring the architecture of open source applications. As I said in my lightning talk at PyCon, the aim isn’t really to explain the internals of Hadoop, Parrot, and Mercurial (though I think that’s worth doing). The real aim is to teach people how to think about software architecture by showing them how architects think. We’re hoping to have chapters in for review by November, and the book out this time next year.

Book #3 is an illustrated children’s book about the universe, life, science, and global warming. I’ve had some good feedback from the editor who handled my last children’s book, but most of the work is still in front of me.

Projects I’m not working on:

Government 2.0: I enjoyed working on open data/open government projects with my students last term, but I couldn’t find any faculty at U of T willing to keep it going. I could have found Gov 2.0 stuff for CSC302, but I thought open source work would be better for them.

Two novels and half a dozen short stories. I enjoy writing fiction, but it feels like an indulgence, and I keep pushing it aside to do “serious” stuff. I’m sure that when I’m seventy I’ll regret having done that, so I hope to spend one hour a day writing fiction once I start full-time on Software Carpentry.

Jazz: I haven’t touched my sax since this time last year—it may be vanity, but I’d rather not play at all than play badly. Maybe when my daughter’s a little older…

Exercise: yeah… exercise. Maybe I’ll get my bike back on the road this week…

Basie, Government 2.0, Making Software, Research, Teaching, Uncategorized

It’s a sunny Sunday morning in Atlanta, and I’m on my way home. I came down Thursday to:

1. Raise money for Software Carpentry.
2. Get people excited about Basie.
3. Get people excited about UCOSP.
4. Talk with Georgia Tech‘s Mark Guzdial about computer science education.

#4 actually happened first. Mark picked me up Friday morning; we chatted for a while, then I spent an hour with some other faculty before giving my evidence-based software engineering talk. It was fun, and I came away from my discussion with Mark with half a dozen leads to follow up.

#1 is most important to me personally—I really want to spend a year upgrading the course after I leave U of T at the end of this term—but I didn’t have much luck. The people I spoke to were sympathetic, but it’s been a hard 18 months for everyone financially, and there are a lot of other good causes clamoring for attention.

I put less time into #2 than I probably should have, but still got some good feedback (which I’ve posted on the Basie blog). Long story short, if we can make Basie faster and provide a Trac-to-Basie migration tool, our prospects are good.

I wasn’t thinking of #3 (UCOSP) when I proposed my talk, but it’s what people were most interested in. Several students and professors said that they would like to be involved; the trick now is to find money to hire a half-time admin to take care of fundraising and organization.  If you have $35K you can spare, please let me know. (And my slides are up if you’re interested.)

The best part of the trip? Talking to people I’ve only ever met electronically, or haven’t seen since my last PyCon eight years ago. Some of the discussion was about programming, but not a lot (since I don’t actually program any more). Mostly it was about kids, careers, and the meaning of life—all the catching up you do with people that you really wish you got to see more often.

It’s a sunny Sunday morning in Atlanta, and I’m on my way home…

Later: video of my lightning talk on Friday evening about yet another collaborative O’Reilly book (this one on software architecture) is available at blip.tv — check about 9 minutes in.

Basie, Python, Software Carpentry

We are pleased to announce the release of Version 0.6 of Basie, a lightweight software project portal built on Django and jQuery. Basie is designed to replace Trac and DrProject; its main features are:

• Multiple projects per forge
• Role-based access control
• Pluggable user account management
• Per-project wiki with standard WikiCreole syntax
• Subversion repository browser
• Per-project mailing list and IRC channel
• A simple ticketing system
• Milestones and calendaring
• Cross-component search
• Dashboard summarizing project activity
• Web-based administration
• REST API for web services

Thanks to the whole team for all their hard work.

Announcements, Basie, Python

Yesterday, I asked if anyone was building something like Fossil using a NoSQL database.  In response, someone named Pete (who didn’t leave a real email address) asked, “Why NoSQL? This is a perfect application for an SQL database.”  Respectfully, I disagree, but my reason will take a little explaining.

Over the past 16 months, several waves of students have been working with me on a replacement for Trac called Basie. Like Trac, Basie is meant to be a minimalist software forge: it combines version control, ticketing, wikis, the like into one package, but is much simpler than the open source and commercial forges that Jordi Cabot and I surveyed. Unlike Trac, Basie is built using modern web tools (Django and jQuery), and supports multiple projects per forge and per-project mailing lists out of the box.

We’ve dealt with quite a few design challenges while building Basie, and have a few more piled up to worry about in January—see, for example, Ian Lienert’s post about deleting vs. hiding, or Andrew Schurman’s look at why integrating with IRC is hard (short answer: channel management). Many of the hardest challenges, though, have a common root cause: relational databases don’t support polymorphism. Take tagging, for example: in order to find all items tagged with “upgrade”, we have to issue and aggregate multiple queries, because the entities that have been tagged are stored in separate tables. Not only does this hurt performance and make the code (much) harder to understand, it also means makes plug-and-play extensibility a lot harder, since anyone who wants to add a new module to Basie has to either edit the tagging code to reference that new module, or wrestle with some not-yet-implemented registration and callback mechanism that moves the grief out of Django’s ORM (where it belongs) and into pure Python code.

We ran into the same problem building the status dashboard, where we wanted one module (the dashboard) to be able to ask questions of others without knowing exactly what those others were or how they were implemented. This is trivial in a programming language that provides polymorphism (as almost all modern languages do), but there’s no standard, straightforward way to do it with SQL.

Hence my interest in NoSQL databases. What they’re explicitly doing is setting aside the “rows and columns” model in favor of—well, in favor of a bunch of different things, depending on which one we’re talking about. But in doing that, they’re sort-of-accidentally making a lot of other innovations possible. After all, if PostgreSQL came out with some kind of polymorphism extension, I probably wouldn’t use it, because I wouldn’t want to be tied to any one relational database. If I choose to use MongoDB or CouchDB, though, I’m committing to a single-source solution anyway, so why not make full use of everything it offers?  Simon Willison (who knows much more about all of this than I do) made a similar point in July when asked how hard it would be to get Django running on top of MongoDB:

I remain sceptical of projects that attempt to map Django’s extremely relational ORM to non-relational backends. Why would you want to do this in the first place? Presumably because you want to use parts of the Django ecosystem – in particular the admin, generic views and pagination – with a different persistent store.

I would argue that you don’t want a ORM backend for MongoDB – instead, you want the admin, generic views and pagination to work with alternative storage mechanisms. Instead of depending directly on the ORM, they should make use of an abstract interface which can be mapped to the ORM but can also map to other types of persistent store.

See also this article from John Nunemaker.

Basie

We’re about to release version 0.6 of Basie, a replacement for Trac built on Django and jQuery. Before we dive into work on 0.7 at the beginning of January, we’d welcome some code and design reviews from experienced Django and jQuery developers. We have some nice coffee mugs to send as thank-yous…

Basie

MarkUs, a Rails-based tool for marking student programming assignments, just released Version 0.5. MarkUs aims to provide the convenience and flexibility of pen-and-paper marking over the web; it will be deployed this fall at the University of Toronto, and is one of the eight cross-country open source projects we’ll be working on starting in—um, really? Just one week from now?

And speaking of the fall, Bill Konrad has posted some ideas about what’s next for Basie. If you have preferences, we’d be grateful for your input.

Basie, Teaching

We are very pleased to announce the release of Version 0.5 of Basie, a lightweight software project forge built on Django that borrows many ideas from Trac and DrProject. As this screencast shows, its features include:

• Multiple projects per forge
• Role-based access control and pluggable user account management
• Subversion repository browser
• Cross-component search
• Per-project wiki with standard WikiCreole syntax
• A simple ticketing system
• Web-based administration
• REST API for web services
• Available in English, Bulgarian, Chinese, French, German, Hungarian, and Japanese
• Graphical dashboard showing project statistics and history (in beta)

Basie is still very much a work in progress (we picked “0.5″ as a version number because we figure it’s halfway done ), but we are very pleased with how it’s coming along, and invite anyone who is interested in trying it out, or helping with its development, to get in touch.

Basie

Now that the Software Carpentry course is over, it’s time to get back to planning projects for the fall, and there’s good news on two fronts.  The first is that we had a very productive meeting on Tuesday about projects using the City of Toronto’s data.  The CUPE strike may mean that we start off using canned data instead of live feeds, but lots of people are willing to put in time to help our students figure out what’s worth building and how to build it.  Ten undergraduate and two graduate students are already signed up; on past experience, we’ll probably have 15 and 20 respectively by the time we start work.

The second piece of news is about the cross-Canada undergraduate projects. We now have interested faculty at:

• University of Alberta
• University of Calgary
• Carleton University
• Concordia University
• École Polytechnique de Montréal
• Memorial University
• Queen’s University
• Université de Sherbrooke
• Simon Fraser University
• University of Toronto
• University of Victoria
• University of Waterloo
• University of Western Ontario

and several have already started recruiting students.  Our project list has also grown to include:

• Basie
• Eclipse IDE for Education
• Elmcity
• MarkUs
• RoboCup
• WikiDev

I’m stoked   For more information about both sets of projects, please see the combined course page. And if you’d like to get involved, please give me a shout.

Basie, Teaching