One of the students who’s going to be working on Hippo this fall sent a message to the group asking for everyone’s MSN IDs, so that the team could set up group chats. That got me thinking about how many different ways there are to communicate electronically these days, and about how to use them to manage a small, part-time software development team.

First, some definitions. Targeted communication is aimed at a particular receiver, or set of receivers, while untargeted just puts content in a well-known place for anyone to read. Email and instant messaging are generally targeted, while newsgroups and wiki pages are untargeted; an always-on IRC channel is somewhere in between. Archived communication “sticks around”, i.e. can be re-read or searched at a later date, while transient communiation evaporates. As with the previous category, the dividing line is not a clear one—newsgroups feel archived, but most servers throw away old articles after a time. Similarly, instant messaging feels transient, but many IM clients now allow you to record conversations. Finally, content that is pushed reaches the recipients without the recipients having to do anything; email is once again the classic example, but weblogs are becoming popular as well. In the pull model, readers have to go out of their way to get messages, i.e. go to a particular web site to read new messages in a forum.

Now, small software development teams want to be able to:

1. ask teammates for help, or request feedback on ideas;
2. report progress, or share things you’ve discovered;
3. schedule meetings;
4. hold meetings (rather than getting together face-to-face); and
5. socialize,
6. all without being flooded.

#1 wants to be targeted to team members, archived, and pushed. (Requests for help don’t actually need to be archived, but answers do.) #2 is similar, though it’s OK for progress reports to be pulled rather than pushed, so long as everyone pulls them eventually.

#3 can be transient, but should be pushed to everyone you want in the meeting. #4 is just a bad idea—the Winter 2003 students found that IM is a very inefficient way to hold meetings. (Check with them if you don’t believe me.) On the other hand, IM is great for #5 (although as Joel Spolsky says, running an IM client at all times is guaranteed to lower your productivity). As for #6, that comes down to the project admins putting spam protection in place, and project members playing nice.

So, with that lengthy preamble out of the way, how should team communicate?

• Use the email lists for questions and short answers, especially when they are time-critical (i.e. someone needs to know how to do XYZ right now).
• Add content to the project web site for progress reports, longer answers, discoveries, FAQs, how-to’s, and so on, but post a note (including a URL) to the project mailing list to let people know when new content is available.
• By all means use IM for socializing, but if you spent ten minutes on IM explaining XYZ to someone, please, copy the conversation, paste it into a page in the project’s web folder, and let everyone know it’s there.

Hm… I’ve left something out here. In fact, I’ve left out the most important point of all:

USE YOUR PROJECT’S ISSUE TRACKING SYSTEM.

(Yes, I actually do think it deserves capital letters.) An issue tracking system is a project’s shared to-do list; it’s used to record bugs, features that need implementing, action items (such as “prepare a note for students on how to communicate”), and so on. What’s more, it records them in a structured way, and that’s what makes issue tracking systems so useful. Email, wikis, blogs, and so on are the “goto” statements of electronic communication: since you can put/send data just about anywhere, there’s no inherent rhyme or reason to traffic and content. An issue tracking system, on the other hand, imposes rules on format, state transitions, and so on, just as “for” loops and “if/else” statements impose order on goto’s.

Experience (mine, anyway) shows that imposing order at the outset is more efficient, and more helpful, than trying to reverse engineer it after the fact by building ultra-smart search engines. Experience also shows that if teams start using an issue tracking system early in the project lifecycle, when deadline pressure is low, they’re more likely to use it well by the time deadline pressure intensifies—i.e., by the time they really need it.

So, how good is your software team? Joel Spolsky (who runs a
company called Fog Creek, and
writes a weblog that
everyone in the software industry either reads, or ought to) has a 12-question
test to help you measure how well your team is doing. How do the
student teams hosted on Pyre
do?

So, that leaves Helium with 6/12, and
Memview with 5/12.
Let’s see how much better we can do this fall…

Well, here they are: this summer’s favorite links.

Where has the summer gone? We’ve written a lot of software (and documentation), but there’s still a lot to do before Helium (er, sorry, Hippo—we’re renaming it, for reasons that are too silly to go into) is ready to deploy.

Eleven undergraduates from the Department of Computer Science at the University of Toronto will be carrying on with the project this fall. Between now and September 9, we have to decide what we want them to work on. In order to do that, we have to resolve some technical issues that we’ve been deferring.

For example, Hippo includes mail management, so that projects can have mailing lists. We’d like to make use of whatever SMTP server our host is running, but that means we have to find a way to get messages into Hippo. We also have to put some kind of spam filtering in place.

However, we also want to keep Hippo unilingual if at all possible. It would be fairly easy for our existing programming team to write a Python script (or even a shell script) to feed Hippo mail, but if we do that, I’m afraid we’ll be falling into the same trap that SourceForge did. Last time I checked, SourceForge depended on five (5) different languages (PHP, Perl, Python, Bash, and C), plus more than a dozen major third-party packages and many smaller libraries. Getting all this stuff installed is hard; keeping it all in synch is harder; and finding programmers who know XYZ when it breaks is a continuing headache. (“Sorry, dude, if this bit was written in Perl, I could help you…”)

So why not write the necessary glue in Java? Performance. Write a little shell script that runs a simple “Hello, world” script in Python a thousand times, and see how long it takes to run. Now do the same in Java. Make sure you have other applications running on the box, so that you’re getting realistic numbers, rather than never-to-be-exceeded everything-in-memory peaks. On Pyre (the Linux box that’s hosting Helium’s development), the difference is about 4:1. It was less (about 3:2) on my souped-up development box at work, but only when I wasn’t running anything else. As soon as the VMs were fighting with other applications for space, Java lost: badly.

We have several ways to deal with this. One is to bite the bullet and write the smallest possible shell/Python script to move mail from the SMTP server to (for example) a scratch directory, then have a long-running Java daemon do everything from there. (I’m pretty sure that Java will match Python’s performance on the actual processing, though I don’t yet have any data to back that up.) Another would be to throw up our hands and say, “OK, Java’s been fun, but this is the N’th time (where N is approximately 3) that we’ve found a case where Python would have been easier, so maybe we should start over.” Needless to say, I’m less than excited about that…

I’m sure there are other possibilities; I’m also sure that we’re not the first development team to bang our heads against this particular wall. As always, pointers and advice are welcome…

A few months ago, Carlos Perez blogged about someone else’s claim to have built a mildly complex web app using Ruby in just two months. Perez argued that one reason Ruby-based development was so much faster than Java-based development would have been was:

1. Java web app frameworks typically use lots of configuration files.
2. These are effectively dynamically-typed (i.e. checking is done at runtime, rather than during a compilation phase).
3. If you’re going to use dynamic typing, you’re better off going all the way, rather than trying to weld dynamically-typed configuration onto a statically-typed language.

Perez offers no evidence to back up his argument, but it’s an interesting one nonetheless. Hibernate, Tapestry, Tomcat itself, and many other current-generation Java tools require XML configuration files that are so complex that users are effectively doing bilingual programming, without having a debugger for one of the two languages.

In contrast, dynamic languages like Ruby and Python allow users to type in complex data structures directly, and make it easy to include calculated values. This is (in my opinion) the main reason why SCons (a build system written in Python) is easier to use than Ant: one language, with a debugger, and you don’t have to jump through hoops to say “for each” or “only do this if A and B are true, but C isn’t”. Even Ant‘s inventor agrees: a few months ago, James Duncan Davidson wrote:

If I knew then what I know now, I would have tried using a real scripting language, such as JavaScript via the Rhino component or Python via JPython, with bindings to Java objects which implemented the functionality expressed in todays tasks. Then, there would be a first class way to express logic and we wouldn’t be stuck with XML as a format that is too bulky for the way that people really want to use the tool.

Now that entire books are being written to decry the complexity of Java application frameworks, and urge us to return to simpler code, it’ll be interesting to see whether more programmers switch to dynamic languages, or whether the continuing backwardness of those languages’ application frameworks drives programmers back to Java and .Net.

In this blog posting from May 2004, Johanna Rothman talks about how easy it is to build a schedule for a small project using just yellow sticky notes. Back in June, I commented on Roy Osherove’s whiteboard-based project management, and asked for links to software tools that were as easy to collaborate with. None were forthcoming, but thanks to Michelle Levesque’s recent experience at VanPy, I now know where to look.

Michelle saw people in several conference sessions using SubEthaEdit to take notes collaboratively. For those who haven’t seen it, SEE allows any number of people to edit a single document simultaneously. It sounds like a recipe for chaos, but if authors are willing to follow a few obvious social rules, it can be tremendously productive. Michelle and Karen Reid are already thinking about having students use it during lectures to take a single, shared set of notes (and about how to prevent the one bad apple in every class from spoiling it).

I’m now wondering whether something like SEE could make electronic schedule construction as easy as whiteboards and yellow sticky notes. I know that wikis can be used this way, but wikis feel like email and turn-based games, while SEE has the zing of instant messaging and real-time games [1]. Schedule negotiation feels like it needs rapid, interactive give-and-take, at least for small projects (a dozen programmers, a dozen months).

So, let me ask my question again: who’s building rapid-fire interactive scheduling and tracking tools using collaborative editing? Anyone? Anyone at all?

[1] Like Homeworld, the best real-time strategy game ever built—socks, but the sequels were disappointing…

Four weeks left until the summer’s work on Hippo (formerly Helium)
winds down, and we’re starting to run into the 10% of cases that make
up 90% of the grief. For example, consider the problem of keeping
track of the relationships between users and projects. In SourceForge and other systems, this is a
simple pairwise relationship: projects are not related to one another,
and users do not belong to groups (other than the groups implicitly
formed by their membership in particular projects).

That’s not good enough for Hippo. Students are naturally grouped
by the courses they belong to; instructors must be able to do
groupwise operations (such as making all students in course C members
of project P) in a single step, or Hippo’s administrative overhead
will be prohibitive. Similarly, instructors must be able to manage
batches of projects at once, so that they can do things like delete
all projects associated with Exercise 3 of Course C with a single
command (after backing them up, of course).

To handle this, we’ve organized projects as a tree, and users/user
groups as a graph. A single root “super project” represents Hippo as
a whole; every other project must have a parent. Similarly, users can
belong to groups, which can also contain other groups (though cycles
are not permitted).

Now comes the tricky part. A user U’s relationship to a project P
is described by one of a small set of roles, such as “observer”,
“developer”, or “admin”. (A fourth role, “unaffiliated”, isn’t
explicitly represented, but is used when U has no other relationship
with P.) Relationships are inherited: if U has no relationship to P,
but does have a relationship R with P’s parent Q, U also has
relationship R with P. This means that if an instructor is an admin
of the project “/csc207″, then she is also automatically an admin of
“/csc207/exercise01″, “/csc207/exercise01/studentFred”, and so on.

Relationships are also inherited via group membership. If U has no
relationship with P, but U is a member of a group G, and G has a
relationship R with P, then U has relationship R with P as well. For
example, if a student is a member of the group “csc207Students”, and
“csc207Students” is an observer of “/csc207″, then the student is an
observer of “/csc207″.

Seems pretty simple—at least, it seemed pretty simple to
us. However, inheritance means that a user U can have several
different candidate relationships with a project P. For example, U
can be an explicit observer of P, but also be a member of a group G,
which is a developer for P’s grandparent. Which relationship should U
have with P? We decided two and a half months ago to use the
“strongest” relationship: we find all possible relationships via
transitive closure, and if any of them allow U to perform a requested
operation on P, the operation is permitted.

The problem is, that leads to Bug
#36:

Users are allowed to have multiple memberships to a single project,
both through inheritance and through user groups. When a user wants
to change something (ie mail settings), the getMemberships(user,
project) method has to decide which of these memberships to return.
Currently, it returns the user’s strongest membership, based on role.
The problem with this is that if the user’s strongest membership is an
implicit one, or if it is through a user group, the user is unable to
change the settings (unless an explicit membership is created for the
user and project).

Let’s go over that again. Hippo represents relationships between
users and projects using instances of the class Membership. If
membership is implied, rather than direct, then when we look up U’s
membership in P, we sometimes get back an object representing the
relationship between a group that U belongs to (directly or
indirectly) and a project that is a parent of P (directly or
indirectly). That’s fine if we just want to find out whether or not
some operation is permitted, but if we want to change U’s
relationship with P, what do we do? We can’t change that relationship
object, since that would potentially affect other users’ relationships
with other projects.

OK, so we don’t change that object; we add a new one representing
the more specific relationship. That solution fails to address Bug
#37:

Another problem with implicit memberships: users cannot be deleted
from projects (or have their roles changed etc.) when their membership
is via a user group, unless they are removed from the group. Removing
a user from a user group falls under user-user authorization, which
has been deferred.

And on, and on, and on. Representing all relationships
explicitly (no groups, no project tree) would make this problem go
away, but we believe it would make administration much more onerous.
Adopting Unix-style permissions (rwxr-xr–, anyone?) is tempting, but
we worked through that three months ago, and it doesn’t address all of
our use
cases either.

Which brings us to this posting’s title. HP’s marketing literature
describes the product I work on in terms of authentication,
authorization, and access control. Authentication means figuring out
who you are; authorization means figuring out what you’re allowed to
do; and access control means enforcing those rules. There’s a ton of
literature about authentication, and a fair bit in the operating
systems world about access control. By comparison, authorization is a
very poor cousin indeed: you can find detailed descriptions of the
schemes used in particular systems, like Unix permissions and Access
Control Lists (ACLs), but there doesn’t seem to be any “theory” behind
it all.

Authorization seems to be one of those things that is overlooked by mainstream computing, for no good reason. Think about it: parsing and code generation are part of the standard curriculum, but implementing byte code interpreters and debuggers are not; process scheduling is, but linking is not; and so on. I don’t know why some fields are “poor cousins”, while others are so heavily overfarmed that they suffer from the intellectual equivalent of salinization, but perhaps there is scope here for both innovative research, and for new languages or programming models to make some headway.