The Third Bit

It’s time to start thinking about what to add to DrProject this summer. Please vote for any two of the following; I’ll collate and re-post.

1. web-based administration interface
2. user-oriented pages
3. status dashboard
4. manual RSS feeds
5. continuous integration
6. pluggable authentication
7. web services
8. shared bookmarks
9. calendaring

1. Web-Based Administration Interface

Instructors and TAs should be able to do simple things (like add a user to a project) through a web interface. Jeff Okawa prototyped an admin interface in Fall 2005; we’ll need to update that, extend it, and add it in.

2. User-Oriented Pages

This page would show a user all the tickets and timeline entries for all of the projects she belongs to, so that she doesn’t have to cycle through three or four separate views to get a picture of what she’s supposed to be doing. Raheel Ashraf prototyped something like this in Fall 2005; we’d have to re-design from the ground up.

3. Status Dashboard

This is Maria’s work from last summer — one form is a graphical display of what’s in the timeline view for a single project, a second is a graphical display of the user-oriented page (#2), and a third (aimed at instructors) summarizes the status of all the projects in the system, so that problems can be spotted early.

4. Manual RSS Feeds

Right now, each project automatically generates an RSS feed of timeline items; there’s no way for users to inject hand-written entries (like, “W00t! We did it!”). The plan is to have a second RSS feed per project containing the hand-written entries, which would also show up in the timeline RSS; that way, when the latter is disabled (as it usually will be for student projects, because of the authentication problems we’ve discussed before), groups can still feed out selected information.

5. Continuous Integration

CI is the practice of rebuilding the software, and rerunning the tests, every time someone checks something in. It’s hard to do in the general case (how many different ways to build, and report build status, can you think of?), but if we can handle simple Ant and Make cases, that’s probably enough…?

6. Pluggable Authentication

We decided a year ago to make the underlying Linux system responsible for user provisioning (i.e., accounts and passwords). That’s now proving awkward: in order to get the Foodbank project collaborators access to the source code and tickets, for example, we have to set them up with guest accounts on CDF. A pluggable auth system would allow us to add our own accounts as needed (but would open up a big can of very squiggly worms).

7. Web Services

Providing either a REST or SOAP API to DrProject, so that it can be scripted remotely, should be relatively straightforward. Throttling usage, so that students don’t do silly things, will be more of a challenge…

8. Shared Bookmarks

A del.icio.us-style bookmarking system would allow students working on parallel projects to save useful links (pointers to relevant parts of the Java API, for example). Dunno how much students would actually use it…

9. Calendaring

Finding times to meet, and keeping track of due dates, is one of the most tedious parts of student team projects. A simple calendaring system that would let students mark off when they’re available, and when assignments are due, would be useful — I question, though, how much they’d actually use it.

DrProject

OK, I’m a little late hearing about this, but: there’s a company in the States called SeaCode that thinks it’s figured out how to get the cost-effectiveness of offshore programming without the management hassles of an eight-hour time difference. Their solution? Stick a bunch of programmers on a boat, and anchor it just outside the international line due west of Silicon Valley.

Gives a whole new meaning to the phrase “software piracy”, doesn’t it?

Uncategorized

You can tell book publishers are feeling insecure because they’re trying to sell us insecurity. O’Reilly books used to be called “Learning This” and “Programming That”; now, we have “Missing Manuals” (what does the geek in the cubicle next to yours know that you don’t, and is it going to cost you your job????) and “Hacks” (caution: you must be wearing really cool sunglasses in order to appreciate this book). And the frequency of extreme sports anecdotes just goes up and up and up (I must be cool, because I kayak—buy this book, you worthless worm, and maybe some of it will rub off on you).

Who do I talk to about this? Who do I write, call, email, or message to say, “It isn’t working”?

*sigh*

Books

I spent some time yesterday chatting with Koushik Sen, a graduate student at the University of Illinois whose work has been picking up prizes. Koushik’s “concolic” technique combines concrete and symbolic evaluation: basically, he uses program analysis to identify paths through the code, then works backward to generate unit tests that cover those paths efficiently. The combination is effective enough to uncover previously-unknown bugs in well-tested code from NASA, Sun, and elsewhere; you can download his tools (in binary form only, not source) from his web site.

Koushik’s work is the latest snowball in a growing avalanche of almost-ready-for-prime-time code quality tools. Andreas Zeller’s book Why Programs Fail describes a bunch, as do two papers in the latest IEEE Transactions on Software Engineering (“On the Automatic Modularization of Software Systems Using the Bunch Tool”, by Mitchell and Mancoridis, and “CP-Miner: Finding Copy-Paste and RElated Bugs in Large-Scale Software Code”, by Li et al). It seems like static and dynamic analysis are about to hit some critical crystallization point, just as testing did a few years ago when JUnit unleashed a wave of ever-more-sophisticated testing aids that programmers actually used. Translating jCUTE, Bunch, CP-Miner, and Zeller’s tools into “download and drive” plugins for Eclipse and Visual Studio coul be a very cool area to be in for the next three or four years.

Research

I’m having trouble printing PNG images with (supposedly)
transparent backgrounds, and I’m hoping one of this blog’s readers can
help. Here’s one of the images from the Software Carpentry course:

[image not available]
If I open it with the Windows Photo Printing Wizard (on XP), it
shows up as dark blue lines on a faint blue background; when I print
it on a black & white printer, it comes out as black lines on a
white background.

Open it directly in Firefox 1.5.0.1 (also on XP), and it displays
as dark blue on white. When I print it from Firefox, though, I get
dark gray on near-black—sort of, but not quite, a negative image of
what I actually want.

If I create a vanilla HTML page:

<html>
<body>
<img xsrc="http://www.third-bit.com/swc2/lec/img/py02/negative_indices.292x83.png" mce_src="http://www.third-bit.com/swc2/lec/img/py02/negative_indices.292x83.png"/>
</body>
</html>

and open that in Firefox, I get the same behavior. Open that page
in Internet Explorer 6.0.2900, however, and I get dark blue lines on a
faint blue background; printing from IE gives me dark lines on a faint
gray background.

So, what am I doing wrong?

Uncategorized

We’ve run into a rather annoying bug in DrProject. I’ve been keeping some brief notes in this ticket, but I’ll try to organize my thoughts better here.

First, some background. DrProject is run as a cgi script under Apache and is backed by PostgreSQL. Every so often (almost daily), we notice that a number of python/postgres processes are hung:

www-data 32016  0.0  1.1 18112 12344 ?       S    06:24   0:02 python2.4 drproject.cgi postgres 32017  0.0  0.6 17880 7148 ?        S    06:24   0:00 postgres: www-data drprojectdb-svn 127.0.0.1 idle in transaction www-data 32203  0.0  1.1 18112 12344 ?       S    06:31   0:02 python2.4 drproject.cgi postgres 32204  0.0  0.6 17880 7160 ?        S    06:31   0:00 postgres: www-data drprojectdb-svn 127.0.0.1 idle in transaction www-data 32279  0.0  1.1 18112 12344 ?       S    06:38   0:02 python2.4 drproject.cgi postgres 32281  0.0  0.7 17916 7476 ?        S    06:38   0:00 postgres: www-data drprojectdb-svn 127.0.0.1 idle in transaction www-data 32340  0.0  1.1 18112 12340 ?       S    06:44   0:02 python2.4 drproject.cgi postgres 32341  0.0  0.7 17916 7316 ?        S    06:44   0:00 postgres: www-data drprojectdb-svn 127.0.0.1 idle in transaction www-data 32410  0.0  1.1 18112 12344 ?       S    06:52   0:02 python2.4 drproject.cgi postgres 32411  0.0  0.6 17880 7148 ?        S    06:52   0:00 postgres: www-data drprojectdb-svn 127.0.0.1 idle in transaction www-data 32466  0.0  1.1 18112 12344 ?       S    06:57   0:02 python2.4 drproject.cgi postgres 32467  0.0  0.7 17916 7308 ?        S    06:57   0:00 postgres: www-data drprojectdb-svn 127.0.0.1 idle in transaction www-data 32533  0.0  1.1 18112 12344 ?       S    07:04   0:02 python2.4 drproject.cgi postgres 32534  0.0  0.6 17880 7160 ?        S    07:04   0:00 postgres: www-data drprojectdb-svn 127.0.0.1 idle in transaction ...

As these processes build up, eventually the number of active postgres connections exceeds the number of allowable non-superuser connections, requiring us to manually kill these processes.

I have been unable to find a sequence of steps to reproduce the frozen processes, but I’ve been able to run a few short tests when it has occurred:

• When the process hangs, it is due to a long timeline query (ie, 30+ days back)
• You can access other sections of DrProject when the timeline is locking up (ie, the Wiki)
• The DrProject log for the hanging request cuts off while the timeline is gathering events for the Ticket system. Note that there are 5 subsystems that the timeline gathers events for, and the Ticket system is the last of these.
• Multiple requests for the same URL will cause the process to hang at the same point during collection!

After some time, DrProject will stop locking up for these timeline requests and the service returns to normal, albeit with a number of hung processes/connections on the server.

I tried an experiment whereby I inserted a manual gc.collect() call into the timeline code. I verified that with the manual cleanup, the timeline would process normally, and that without the cleanup, it would continue to hang (I switched between these two multiple times, so as to rule out an accidental solution )

Continuing in this vein, I started to investigate whether there were any objects not being collected properly (in particular, leftover database connection objects would worry me). Sure enough, a number of PooledConnection objects are classified as uncollectable by the Python garbage collector. In the simplest form, the problem seems to be isolated to the ‘select’ function in our generic Record class. This class is the base class in our custom Object/Relational mapping framework, and implements the basic SELECT functionality for every object. The following example shows the problem:

from drproject.env import Environment from drproject.project import Project import gc gc.set_debug(gc.DEBUG_LEAK)  env = Environment('/tmp/drproject') for obj in Project.select(env, 'All'): print str(obj)

Which results in the following output:

... <Project name='All'> ... gc: collectable <EntryPoint 0x1256f50> gc: collectable <dict 0x125c030> gc: collectable <tuple 0x1256e70> gc: collectable <list 0x434fa8> gc: collectable <cell 0x43f770> gc: collectable <cell 0x43f870> gc: collectable <cell 0x43f710> gc: collectable <function 0x41c630> gc: collectable <tuple 0x1251cd8> gc: collectable <tuple 0x43f510> gc: uncollectable <PooledConnection 0x434e18> gc: uncollectable <dict 0x4a5ed0>

Note: The dict that is uncollectable is actually the __dict__ object for the PooledConnection that is uncollectable.

My working theory is that during a long timeline request, these uncollectable PooledConnection objects accumulate to the point where Python will hang unless a manual collection is done. This fails to explain, however, why the timeline is locking up in rare occasions, and not all the time.

My next step will be attaching GDB to one of the frozen processes to see exactly what is going on. As I’ve never done this before, it should be a good learning experience

— Sean Dawson

DrProject

Never mind telecoms, internet, and television; the real convergence story is the way that real and fictional worlds are becoming ever more closely intertwined: you will soon be able to earn points toward World of Warcraft gold on your credit card, and Creative Home Engineering will build a secret passage in your home today. (Both links courtesty of Bruce Schneier‘s blog.)

Update: Sean Dawson tells me that the cute little gnome on the credit card is Joi Ito, who is the leader of Sean’s guild. Smaaaaaall world.

Uncategorized

This article, from Jon Udell, describes how he used online services to highlight a section of trail that he’d cleared of trash. He sent the map, along with a letter, to his local newspaper, to encourage others to pick up where he left off. Things like this make me wonder whether online connectivity will, in the long run, favor the civic-minded: people who don’t care about things like this won’t be affected much one way or the other (they’ll just continue to consume), but people who do will be able to make ties with each other more easily. Mumble mumble preferential evolution mumble game theory mumble…

Udell closes with a quote from Gibson: “The future is here, it’s just unevenly distributed.” I think another would be just as appropriate: “The street (in this case, the hiking trail) finds its own uses for things.”

Uncategorized

New Scientist is featuring a story from Denmark that the appearance of photosynthetic life may have led to the formation of continents (or at least aided it). Sadly, two of the three Google Ads that came up when I viewed the page were selling creationist literature…

Uncategorized

A report from Microsoft Research called 2020 Science got a lot of press this week: Nature seems to think it’s the biggest story of the year so far, and The Economist gave it three full columns. Sadly, amidst the gush about how computers are revolutionizing science, no one mentions that most scientists have no idea how reliable their programs are—in fact, most scientists don’t even know how they would figure that out [1,2]. If someone submitting a paper to Nature said, “We didn’t calibrate the equipment, we didn’t write down the settings, and we have no idea what the error bars on our graphs should be,” their work would be bounced without a second thought. Unless computational scientists decide to live up to those standards, the “revolution” that 2020 Science describes will be a long time coming.

[1] “Where’s the Real Bottleneck in Scientific Computing?”

[2] “Computational Science Demands a New Paradigm”.

Research, Software Carpentry