The Third Bit

Yet another interesting and useful post from the prolific Mike Gunderloy.

Uncategorized

Now that winter has finally arrived, the StreetKnit project needs you more than ever.

Uncategorized

I spent some time last week with a group of medical imaging researchers who manage their work with Excel.  They explained that it’s almost ideal for their needs, since they can use cells to organize their experimental results, diagrams, code snippets, and whatever else they like in whatever free-form manner best suits the problem at hand.  Unlike a web browser, Excel can do simple calculations and re-draw their charts; unlike a database, they can merge cells to create irregular structures if that’s what the problem calls for.

The two things they can’t do are:

1. Merge independent edits.  Excel uses a closed-source binary format, so all version control tools can do is say, “These two things are different.”  (I’ve complained about this before.)
2. Interact with their data over the web (more particularly, share live documents with colleagues).

A few years back, scientists in the US national labs were working on an uber-tool that would do all of this and more.  That effort is still going on, but (a) it seems to have slowed down, and (b) it’s a big-hammer approach.  I’m wondering if there’s a role here for an in-browser spreadsheet whose data format is diffable and mergeable.  It’d be useful for much more than laboratory science: course grades spring to mind, as do address lists and a host of other applications…  Anyone want to be rich, famous, and popular?

Uncategorized

Visits to the Software Carpentry site were down quite a bit in December; I had expected a drop because of the holiday, but not this large.  On the bright side, 47 students have signed up for the course at U of T this term; about 1/3 are from Computer Science, while the rest span Civil Engineering, Ecology, Physics, Ecology, and the life sciences.

Software Carpentry

Courtesy of David Scannell, here’s a quick rundown of how to create a new table in DrProject‘s database. (Yes, I know, we should convert to SQLObject or SQLAlchemy; if you have time to do the work, please let me know…)

Suppose I want to create a table called Demo with three fields:

• Named: string
• ID: integer (primary key)
• TicketID: Integer (foreign key)

We’ll assume that all work is being done in a drproject/demo directory, that you have __init__.py set up, and you have modified setup.py file to recognize the new package.

Step 1: Create the Model

A. Create a file call model.py.

B. Import the following:

from drproject.db.record import *         # Base for all database modelling

from drproject.ticket.model import Ticket # Because we want a foreign key

C. Create a new subclass for record.

Class Demo(Record):

D. Associated this model with a database table (by default the table it gets associated with is the same as the name of the class):

Class Demo(Record):   _table='demo_table'

E. Now add the columns to the table:

Class Demo(Record):

_table='demo_table'

id = IdColumn()                # Creates a self-incrementing id

name = TextColumn()            # String column

ticket_id = ForeignKey(Ticket) # Makes this a foreign key to the Ticket Table

F. Make the ID column our primary key:

Class Demo(Record):

_table='demo_table'

id = IdColumn()

name = TextColumn()

ticket_id = ForeignKey(Ticket)

_key = (id,)

Done.

Step 2: Create the System

A. Create a file called api.py.

B. Import the following:

from drproject.core import *

from drproject.env import EnvironmentSetupParticipant # For database creation

from drproject.demo.model import *                    # Our database model

C. Create our EnviromentSetupParticipant subclass.

Class DemoSystem(EnvironmentSetupParticipant):

D. Associate the table that this new system will work with. In our case, it is the demo table.

Class DemoSystem(EnvironmentSetupParticipant):

database_tables = (Demo,)

E. Give our system a name and version number:

Class DemoSystem(EnvironmentSetupParticipant):

database_tables = (Demo,)

version_propery = 'demo_system'

version = 1

Done.

Step 3: Finishing Off

Now, when we create a new environment, Drproject will go over all the EnvironmentSetupParticipant objects, and create all the database tables each one is associated with. Process occurs in the method __init__ of the class Environment in the file drproject/env.py.

DrProject

This short essay by Bruce Schneier is an excellent little summary of what has to be done to make software more secure.  In the wake of this morning’s announcement that systems at several major retail chains were hacked as far back as 2003, I’d really like elected officials and other policymakers to read it…

Uncategorized

Collections of essays have been around since Montaigne — my copy of Orwell’s is practically falling apart from repeated use. Collecting blog postings is a newer literary form, but as Joel Spolsky, Eric Sink, and others have shown, it can be equally worthy.

A new entry in this field is The Open Laboratory: The Best Writing on Science Blogs 2006, which was put together in just a few days, and is available through Lulu.com. The essays are all still available online, but I’ve ordered hardcover — I still find that makes folding down corners easier .  And if this topic interests you, check out the North Carolina Science Blogging Conference on January 20.

Uncategorized

As Craig Andera observed last March, there’s no such thing as “just one application”. Every real program is surrounded by ancillary tools to pre- and post-process its data files, monitor its status, connect it to other programs, and so on. Six months after deploying DrProject in classrooms at U of T, here’s a snapshot of what else our doughty system administrator has to mess with when he’s messing with DrProject:

• /home/software/rc to restart various instances (one per course) when the server goes down and comes back up. This file is executed upon boot by being symlinked in as /etc/rc2.d/S99zlocal.
• /usr/local/bin/stopstart, a script he wrote to stop and re-start instances. It would not be necessary to do this in such a “user-friendly” way if DrProject didn’t crash occasionally (due to a concurrency bug we’re still trying to track down). Once that’s solved, it would suffice for the sys admin to be the one to restart wedged instances, because that would be rare again. On the other hand, Apache still comes with a user-friendly stop/start script…
• /usr/local/bin/stanleywatch is another script that periodically checks the URLs of running instances and notifies us if they’re down. It takes its name from the server’s (stanley.cs.toronto.edu). We’ll keep this even when DrProject is rock solid—in fact, it’ll probably be even more important then.
• pingdrproject is a script driven by a cron job that accesses a mail URL in each instance every few minutes so that mail will be pushed out. It should be possible to get this to be triggered by the arrival of new mail instead, or (better yet) DrProject could supply a program which the sys admin could set as a pipe-type target in the .forward file or /etc/aliases entry. Instead of the /etc/aliases entry or .forward having to put the mail message in a file in the right directory in the right format, and then strobe DrProject so that it actually processes the new mail, there could be a program which receives a raw e-mail message on standard input and just processes it right there. That is, the arrival of new mail would itself trigger the processing, and there would be no issue of maildir versus spooldir and so on.
• /etc/apache2/sites-enabled/000-default, which is the standard Apache config file for the primary virtual host on that machine. It maps the URL https://stanley.cs.toronto.edu/svn/course-id/project-id to run the DAV Subversion module for the directory name /home/courses/course-id/drproject/repos/project-id, and the URL https://stanley.cs.toronto.edu/course-id/drproject/project-id/wiki/PageName to be relayed to the DrProject server process listening on port 4003 with pathname /project-id/wiki/PageName. It also has other mappings for the normal server web pages and such, which don't change as courses come and go.

It’s more to install, configure, test, and keep consistent than we’d like; one of the goals of Engenuity‘s work in the coming weeks will be to simplify this as much as possible.

DrProject

The Fifth International Conference on Pervasive Computing (you know, the stuff that watches your every move?) is being held in Toronto this May.  Funnily enough, the web ad for it also mentions that the 2007 World Horror Convention is being held at the same venue.  Coincidence?  We think not…

And the editors of the Chicago Manual of Style have weighed in on the highly-charged question of formatting emoticons within parentheses…

Uncategorized

Martin Fowler’s Refactoring (which is on just about everyone’s software engineering must-read list) is being translated from Java to Ruby.  Meanwhile, I’m still waiting for a good Python design patterns book…

Uncategorized