The Third Bit

OK, we’re not quite done with tickets yet—at least, not with the issues that tickets raise.

The first one is email notification. In Trac and many other systems, you can specify that you want to receive email every time a ticket that you filed is updated. This is a lot handier than logging in every once in a while and asking the system to display all the tickets you created that have been modified in the last, um, six days? Ten? When did I last log in, anyway?

We took this out of DrProject because we thought there wouldn’t be a need for it. It’s common in industrial-strength systems to open up ticketing to customers and users, so that (a) the number of people who can file tickets is much greater than the number who will be dealing with them, and (b) many of the people filing tickets aren’t “members” of the project in any sense. In contrast, the primary use case for DrProject is lots of little teams, each of which is developing the software primarily for itself. Since DrProject supports per-project RSS feeds (see below), email notification of changes to tickets seemed redundant.

Well, we [1] were wrong. Even on a two-person team, people like to be poked when tickets they care about change (or when new tickets are assigned to them). What we failed to take into account is that students are slicing their time between many different courses; it is very useful for them to have their attention drawn to things as they happen, and email is still the best way to do that.

Turning email notification in specific circumstances back on won’t be too hard. What will be is generalizing it, so that people can (a) specify a rule for when they want notification, and (b) override that rule to be notified (or not) for specific tickets. For example, as an instructor, I might want to set things up so that I’m only notified about changes to tickets assigned to me, and not to tickets I created, since I’m going to be filing dozens or hundreds of tickets over the course of a term to give students work. I might also want to add myself as an observer of particular tickets that I didn’t create, and that I’m not supposed to be working on. It isn’t rocket science, but it’s going to be a grumpy bear to test.

And speaking of RSS: like Trac, DrProject creates an RSS feed (in DrProject’s case, one per project) that summarizes the events occurring in a project. Ticket changes, Subversion check-ins, wiki page edits, and the like are all turned into blog entries. It’s a great way to keep track of what’s going on in projects, particularly if you’re trying to monitor a dozen at once.

Unfortunately, we’ve had to disable the RSS feeds for our undergraduates’ course projects. We can’t allow one team to see another team’s RSS feed, since check-in comments and ticket bodies can contain sensitive information [2]. If we could password-protect the RSS feeds, we would, but there isn’t yet a standard way for blog readers to forward credentials. (And even if there was, we wouldn’t be comfortable with students storing the passwords for their lab accounts on web-based feed readers like Bloglines.) We don’t see any way to fix this one…

Another feature of Trac that we’ve dropped is custom ticket queries. The idea is seductive: allow users to write arbitrary queries against the ticket database, store them, and create links that trigger them, so that other users can run the same or similar queries. This is a complete non-starter in our context: there is no way we’re going to allow students to run their SQL against a shared multi-project database. That said, it would be nice if there was a way to capture and save the settings someone is using to view the ticket database, so that people could share their favorite views on what it contains…

The last [sic] part of the ticketing system we’d like to revisit is the set of actions users are allowed to perform. Right now, you have three choices when you update a ticket: leave it open, mark it as complete, or reject it. No matter what you do, you’re supposed to write a short comment explaining your action (just as you’re supposed to write comments when you check files into Subversion).

This is working well, but doesn’t capture one important use case. The more users a project has, the more likely it is that someone will file a ticket that duplicates one filed earlier. At present, we rely on users to write something like “Duplicates #456″ when the reject redundant tickets; all three of the companies that are thinking about adopting DrProject internally would like something a little more structured, so that they can (for example) search for tickets that were duplicates.

The reason we haven’t done this is that it complicates our representation of ticket states. Right now, states are taken from an enumeration with three values; if we want to add “duplicate” as a fourth state, we have to add another field to keep track of which ticket (or tickets) this one is a duplicate of—fields that wouldn’t be meaningful in other states. Irregularities like this are bad news for maintenance and testing; we won’t add “duplicate” until we can think of a way to avoid this one.

Of course, suggestions are always welcome…

---

[1] This is a royal “we”…

[2] Yes, the world would be a better place if students could share information and learn from one another. However, part of our job is to evaluate their work, and we need to be sure that it’s their work, which limits how much plagiarism sharing we can allow.

DrProject, Teaching

Over the past couple of weeks, I’ve had some interesting conversations with local small software companies about their development process—in particular, about how they collect and manage requirements. It isn’t anything like it is in the textbooks, and I think I may have figured out why.

Here’s one web development company’s process:

1. A technical salesperson interviews customer to find out what the customer wants. For the most part, this consists of checking off items on a spreadsheet: “User self-registration… Mailing lists… Blog feeds… Product overview pages… Product pages… Got it.” There may be a few “custom components” in the mix, but for the most part, the customer’s web site can be assembled by customizing and combining off-the-shelf components.
2. The salesperson estimates the size and complexity of the job on a small/medium/large scale. This selects the scaling factors that are applied to the individual work items, the basic implementation time of which is built into the spreadsheet. The salesperson also estimates the time required to build any custom components, usually by analogy with similar items done previously. What comes out the end is a preliminary estimate of the project’s cost.
3. This estimate is taken back to the customer, who is then told how long it will take to firm up the design (a necessary step toward getting a final cost). If the customer says, “No thank you,” the two sides shake hands and go their separate ways. Otherwise, the company works on a time-and-materials basis to turn the rough estimate into a design detailed enough to form the basis of a contract.
4. The finished design is taken back to the customer—in fact, it is the customer’s property. If the customer wants to take it elsewhere for implementation, there’s another opportunity for a handshake and a farewell; in practice, though, that has never happened. What does happen is that customers try to whittle down the price (which usually results in work items being pushed off, since the one thing customers are usually absolutely firm on is deadline).
5. The web development company then translates the design into code. Usually, the result is a happy customer, but sometimes the customer looks at the finished product and says, “This doesn’t do what we asked for.” This is the point at which it becomes clear that the customer didn’t actually read the design document: either they trusted the web development company, or didn’t have time, or couldn’t make any sense of it. What follows is usually what the diplomats refer to as “full and frank discussions”…

Here are some observations:

1. The development company has to commit to implementation details very early on in order to tell the customer how much things are going to cost (which in turn shapes the customer’s requirements—if you can’t afford it, you don’t need it). This is at odds with textbook approaches, in which you figure out what the customer wants, then figure out how you’re going to build it.
2. The initial spreadsheet isn’t nearly detailed enough to give to developers on its own. Since the customer has already told the development company what it wants, the development company can’t let its programmers go back and re-interview the customer to flesh the spreadsheet out (double cost to the customer, and it would look unprofessional). The technical salesperson who spoke to the customer therefore has to dump everything s/he knows into to the mind of whoever is going to work up the detailed design as quickly and accurately as s/he can.
3. The detailed design is done in terms of features: this page has widgets with these capabilities, that database stores information according to such and such a schema, and so on. Customers, on the other hand, think about what they’re buying in terms of a muddled mix of pages and business processes: pages, because using the web has trained them to think that way, and business processes, because they want the web site to do something. Mapping the customer’s stew to a feature list is hard; mapping the features back to the stew in a way that the customer can understand is close to impossible.

Why does this company do things this way? Because any other methodology would lead to bankruptcy. If you’re IBM or Arthur Anderson, and the job itself is going to take five years and cost forty million dollars, customers will pay for your up-front design time (which may well be six months or a year). Customers will not pay for the up-front on a two-person, six-week web site upgrade (in part because the kind of customer who needs that kind of job done usually isn’t particularly clued in technically). You have to figure out what the job’s going to cost to make the sale; you need to know how you’re going to build it to figure out what it’s going to cost; so you need to turn stew into implementation features much earlier than purists would have you do.

Another interesting observation is the way that this process steers development companies (not just the one whose process is outlined above) in certain directions. The more you standardize your offerings, the better you will be able to estimate costs. Whether it’s web site development, telecoms, or data analytics, small companies doing on-demand work all seem to try to componentize their offerings in order to avoid losing their shirts.

Finally, none of the people we’ve spoken to so far have much time for Extreme Programming. Customers need to know what they’re going to get months ahead of it being delivered, so that they can firm up all their other plans. If purchasing tells marketing that the spec is, “We’re giving them money, and they’re going to satisfy our needs,” someone’s gonna lose their job. The only exception is when the development company has built up a lot of trust with the customer; if the developers have delivered above and beyond several times running, the customer may well say, “Here are the keys, take it away.”

Here are the takeaways:

• In order to succeed in the real world, a development process for small and medium-sized teams has to allow those teams to determine the time and effort a job is going to require very early on. That in turn requires a “leap to implementation”, with all the concomitant risks of focusing on plumbing at the expense of decor.
• Companies want a lightweight, dynamic way to manage requirements. One idea (which we’ve heard twice) is to file a ticket for each requirement, then link those tickets somehow to the “feature to be implemented” tickets that satisfy it. Such a system would also need to allow its users to impose an order on those tickets, so that they could be run off as a single, coherent document. (Don’t groan—what’s important here isn’t the paper, but a sensible linear ordering.) This ties in nicely (we think) with adding parent/child relationships to tickets to help with scheduling (“X can only be started once Y and Z are finished”, and so on).

Thesis topic, anyone?

Research

So that’s tickets taken care of, right? If only… Versioning is far from being the only thorny issue in building a ticketing system. To understand some of the others, it helps to know a little about how DrProject evolved from Trac.

When we first started using Trac in January 2005, each installation of the software managed one, and only one, project: a single ticket database, a single set of wiki pages, and (most importantly) a single Subversion repository. Each installation needed its own stanza in Apache’s configuration file, and users had to be added to each installation separately (Figure 1).

That model made sense for one-of-a-kind open source hobby projects, but broke down in larger settings. Even a small company, for example, would usually have several projects going on at once; setting up and managing each one completely independently would be (to quote a former colleague) “nuthin’ but overhead”. And in a classroom setting, Trac‘s model didn’t work at all, since instructors need to start twenty or more nearly-identical projects every term.

There were several ways we could fix this:

1. Put a facade in front of a collection of Tracs to route HTTP and Subversion traffic to the right place (Figure 2).
2. Partition the database by adding a project ID column to each table, and use Subversion access control files to give each user access to only part of a common repository (Figure 3).
3. Partition the database as above, but give each project its own Subversion repository (Figure 4).

We discarded the first option fairly quickly, since it would make integration (particularly configuring the web server) a lot more complicated than it already was, and would be very difficult to test and debug. Option 2 seemed attractive for a while, but then we stumbled across this scenario:

• Hamdan is a member of project H; Tong is a member of project T.
• So Hamdan has read/write access to /svn/drp/H, and Tong has read/write access to /svn/drp/T. Neither has any kind of access to any other part of the repository.
• Hamdan creates /svn/drp/H/README.txt and adds it to the repository. This becomes changelist 1.
• Tong creates /svn/drp/T/design.html and adds it to the repository. This becomes changelist 2. Tong says, “Huh?”
• Hamdan modifies /svn/drp/H/README.txt, checks in his changes, and is told that he has created changelist 3. Hamdan says, “Huh?”

At first, the non-consecutive numbering of changelists within projects sounds like it would just be an annoyance, but it’s more than that. Suppose we want to display the history of a project: we have to filter out all the changelists that refer to portions of the repository that the user requesting the history isn’t allowed to see. And what happens when an instructor adds start.java to every team’s repository, and checks them all in at once (instead of checking in one copy at a time)? What do we show then? What do we let people revert to?

There’s a much larger point here than just how we divvied up students’ files among repositories. The phrase “postmodern programming” refers to the observation that people don’t build programs any more: they assemble them by re-mixing the work of others. 99% of the bits in the last big product I worked on were bought or downloaded; our part, as complex as it was, was “just” the glue holding it all together. It’s the only way to build large systems, but it means you have to know early on—at design time—what assumptions and limitations are built into the systems you’re going to be assembling. Like other architects, we must always design toward the materials at hand.

Here’s another instance of the same problem. How do you identify tickets? Every system I’ve ever worked with numbers them: as Kronecker said, “God created the integers; all else is the work of man.” That maps pretty cleanly to a database table:

id

summary

author

creationtime

…

But what if you want to store multiple projects in a single database? You can’t use a separate table for each project (well, you could, but it would be a lousy design for reasons that I hope are obvious). What about this:

id

project

summary

author

creationtime

…

Seems like it would work: to get a project’s tickets, just add WHERE project=whatever to the SELECT statement. Once again, though, we have a sequentiality problem: if people are filing tickets in different projects over the same period of time, a particular project’s ticket IDs will seem to have gaps.

All right, why don’t we just use the project’s name, and the ticket’s ID within the project, as a composite key to uniquely identify each ticket? Our SELECTs are no more complicated than they were—but then every other table that wants to refer to tickets has to use a two-column composite as a foreign key, instead of a single-column integer value.

Things like this are why I’m sceptical about the most extreme interpretations of Extreme Programming. “Don’t design it ’til you need it” is great if changes don’t have long-range ripple effects, but in my experience, the harder the problem, the more likely it is to send shock waves through the whole system.

Tags

Here’s another example. Over the summer, Apple Viriyakattiyaporn added tagging to DrProject, so that keywords can be attached to wiki pages, tickets, email messages, and just about everything else. She had two independent decisions to make: how to store the tag values, and how to index them. Options for storing the tag values were:

1. Store each tag separately, one per database record.
2. Store all the tags together in a delimited list (e.g., "bug fix|concurrency|GUI") in a single record.

The first wins hands-down: if she had used the second, searching for things with particular tags would have required us to delve into the stored values, which requires more code than a simple WHERE, and is less efficient.

What about indexing? If we only wanted to tag tickets, the table could be:

project id (text)

ticket id (int)

tag value (text)

i.e., a set of unique triples. But we also want to tag wiki pages, whose unique identifiers are strings, not integers. Oh, and email messages (integer keys). And other things that we haven’t though of yet, that might use some other type (such as dates) as an index.

Choices:

1. a separate table for each module (such as tickets or the wiki);
2. one table for each type of index value (such as string or int); or
3. convert everything into strings.

In the first option, the thing the tag belongs to is implicit in the table: only ticket tags appear in the ticket tag table, for example. In the second and third option, the tag tables need an extra column to keep track of this information.

We hemmed and hawed over this for a while, then finally went with option 3. We thought that everything we’d ever want to index tags by would be representable as text somehow, and options 1 and 2 might require us to add new tables (i.e., change the database schema) when adding new components in the future. Changing the schema is always a Bad Thing, since it means upgrading existing installations; “everything is text” seemed like the lesser of the available evils.

Attachments

One final note on tickets. People often want to attach screen shots, configuration files, and other odds and ends to them. Like Trac, DrProject stores attachments in the file system in an attachments sub-directory of its working directory. attachments has two sub-directories, wiki and ticket, which store the wiki and ticket attachments respectively.

Each of these has sub-directories whose names correspond to the wiki page’s name or the ticket’s number; the attached files are put into these directories. Thus, the attachment attach.txt for ticket #222 would be located at /drp/attachments/ticket/222/attach.txt, while the attachment attach.txt for the wiki page MyWiki would be located at /drp/attachments/wiki/MyWiki/attach.txt.

Attachments aren’t versioned (there’s that word again): if a user attaches a fresh screenshot to a ticket that has the same name as the old screenshot, the old screenshot is lost. What’s worse, attachments aren’t segregated by project, either: if someone attaches a file called pic.png to ticket #22 in the project Venus, it will overwrite the file with the same name that’s attached to ticket #22 in the Mars project.

We’re going to fix this as soon as we can find the time. What we should really do, though, is store attachments as BLOBs (Binary Large OBjects) in database we’re using to store the tickets themselves, or backtrack even further and put everything under version control.

DrProject, Teaching

1) Quintura is a search engine that uses tag clouds as an interface.  Very cool, and suggests a whole family of other tools…

2) TimeSnapper captures your desktop every few seconds all week long; you can then replay the sequence as a movie to find out what you were actually doing.

Uncategorized

Traffic on the Software Carpentry site was up again in October, after a dip in September:

The sys admin at Enthought also cleaned the comment spam out of the Trac they gave me to manage the project.  I have two dozen more minor tickets to file from email I’ve received in the last few weeks (much of it from Germany).  Still waiting for people to start contributing content — any volunteers?

Software Carpentry

Toronto’s first Usability Camp will be held on Tuesday, Nov 14.  From the announcement:

UsabilityCamp is based on the same “unconference” format as DemoCamp and CaseCamp but focused on usability – and it’s FREE! We want to bring the product & industrial design, technology, web and business communities together and engage YOU in conversation about design, usability and user experience.

This is all happening as part of World Usability Day – local activities on a global scale which all take place on November 14th, 2006.

The theme for our Toronto event is Canadian innovation and our featured presenters include innovators from Umbra, Sapient Canada, Autodesk, ConceptShare and the Habitat New Media Lab of the Canadian Film Centre.

Check www.usabilitycamp.org for full details.

Tuesday, November 14, 2006
Registration/networking 6 – 7 pm
Presentations 7 – 9 pm

Announcements

Second ultrasound today — she’s a girl!

Two strong femurs:

and plenty of backbone:

Family

At long last tickets… I still slip sometimes and refer to ticketing systems as “bug trackers”, but they’re much more than that. If version control is a project’s memory of where it has been, ticketing is the to-do list telling it where it’s going. New feature ideas, open questions, requirements—I haven’t found a limit yet to what a ticketing system can be used to remember.

The simplest possible ticketing system is a flat ASCII list of things that need to be done. Store this under version control, so that everyone in the project can edit it concurrently without information being lost, and you’re already better off than you were. I ran several projects in the late 1980s and early 1990s this way, and still sometimes miss the simplicity.

But people want more: they want to know who’s responsible for each ticket, how important it is, when it was created, and so on. They also want to slice the data in other ways to find out which tickets particular people (usually themselves) are responsible for, or which tickets have to be closed before Version 3.2 can go out the door.

Historically, the only sensible way to build systems that needed to perform complex queries on structured data has been to use a relational database. It’s easy to create text files with email-style headers like this:

Author: Alan Turing
Owner: Alan Turing
Date-Created: 1950-07-03 13:29:00
Summary: Write a chess-playing program  As the rules of the game are algorithmic in
nature, surely it must be possible to create a stored program capable of emulating
or surpassing human play?

but sooner or later, you have to write a little Boolean query engine so that you can find tickets created by Alan Turing, but owned by someone else, that aren’t about “chess”. If we were inventing issue tracking for the first time today, open source text search engines like Lucene might make text files under version control viable; as it is, every issue tracker I know of has a database in it somewhere (even Roundup, which breaks with tradition in many other ways).

All right: what does our database need to store? Text fields for the creator and owner, another text field for the one-line summary, another for the body of the ticket… Oh, and date fields for when it was created and last updated, and—am I forgetting anything here? Oh yeah, how important it is, and whether it’s a bug, a feature request, a question, um, better throw “miscellaneous” in there as well… And another date field for when it’s due, and—

At this point, alarm bells ought to be ringing in your head. Jumping straight to implementation is a classic engineering mistake; instead of designing a database schema, I ought to be asking who the intended users are, and what they need to do and know. In one word, I should be thinking about workflow.

I wish I could tell you we did it that way. I wish I could say we wrote out use cases, or at least user stories, and derived requirements from them. What we actually did was look at Trac, CVSTrac, Bugzilla, and a couple of other systems, then ask, “Which of the features in these systems will our students actually use?” When I have to, I justify that methodology by saying that DrProject is supposed to teach students how to use grown-up project management tools, so it makes sense for its features to imitate those of real-world systems. Still kind of wish we’d done more thinking up front, though…

So: our typical user is an undergraduate student familiar with version control, IDEs, and automated builds who has never used web-based project management tools. She’s working in a group of six; they have one term (12 or 13 weeks) to design, build, test, and document a moderately complicated extension to an existing piece of software. Here’s a sample of what she and her team need to keep themselves on track:

1. The team will create roughly a hundred tickets during the term. That’s few enough that we can display one-line summaries of all of them on a single web page.
2. Team members will want to see all tickets (so they can judge where the project is as a whole) and the tickets currently assigned to particular people (especially themselves). They may also want to sort by age (to find things they may have forgotten about), due date, and priority [1].
3. They need to change ticket ownership (i.e., assign tickets to one another). We don’t need to wrap permissions around this: if Jiao mistakenly or maliciously assigns all his tickets to Petra, the team will be able to sort it out.
4. The whole point of integrating the ticketing system into DrProject is to hook it up to the wiki and other systems. To simplify this, ticket descriptions should support wiki syntax, and it should be as easy to link to tickets from wiki pages (and other tickets) as it is to link to wiki pages.

There are lots of other issues as well, but these are enough to highlight two key design points. The first is, once again, versioning: as with source files or wiki pages, we want to keep track of who made what changes to a ticket, and when. Partly, this is to protect users against accidents: if someone erases the body of a ticket, we want them to be able to revert it to its previous state. Keeping history also helps people get answers to questions: if you don’t understand the two paragraphs that were just added to a ticket you’re responsible for, you’ll want to be able to find out who added them, so you can request a clarification.

Versioning tickets brings up all the issues discussed previous in the context of wiki pages. How do we handle conflicts if two people are updating the same ticket at once? We don’t: whoever submits their change first wins, and whoever submits second is told that they are/were editing an out-of-date version of the ticket. The problem hardly ever arises in practice, so we’re not going to rewrite DrProject with everything in verison control, but it still irks.

The second key design point that our abbreviated requirements list brings out is that there are three types of fields in tickets:

• free-form text fields, like the summary and body;
• those with other “obvious” types, like the creation and modification dates; and
• enumerated fields, like priority and owner.

Fields of the first kind are obviously stored as text. Fields of the second kind are easy too: dates are stored as dates [2], integers as integers, and so on. But what about enumerated values? Suppose we want to be able to mark tickets as high, medium, and low priority—should we use strings? If we did, we’d have to tell users what strings were allowed, and check that they only typed in values we were willing to accept. Or we could put a table of strings in our CGI program, so that the UI could display them in a dropdown…but then anyone who wanted to add new priority values would have to edit the code.

The best answer is to store a table of acceptable values that the CGI can read, display, and validate against. Our options are:

• A configuration file: easy for the administrator to edit, but yet another thing that has to be in the right place, and formatted the right way, for the system to work. We’d also have to write yet another function to read and validate the values.
• A database table: not as easy to edit (although a command-line database prompt isn’t much different from a simple editor), but we can grab the values with the same SQL queries we’re using to fetch everything else we need to know about tickets.

DrProject uses a database table with two columns, both text. The first stores the type of the enumeration, such as “TICKET_PRIORITY”; the second stores the actual values, such as “high”, “medium”, and “low”. This means that we can put other enumerations (such as ticket states) in the same table, rather than needing a separate table per enumeration.

But there’s a catch here, one we didn’t spot until relatively late in the day. The sets of enumerations are add-only: once a value has been put into an enumeration, it can never be taken out, since doing so could make the database inconsistent. If you decide you only need “high” and “low” for tickets, for example, and remove the pair (“TICKET_PRIORIY”, “medium”) from the enumeration table, all those tickets currently marked “medium” suddenly have illegal priorities.

No problem: we’ll just write a little script that promotes all “medium” tickets to “high” priority, or check that nothing is using an enumeration value before removing it. But what about the historical record? If a ticket was created with “medium” priority a week ago, should we retroactively change it to “high” as well? Or leave it with a now-invalid priority? Or version our enumerations, so that as we go back in time, our sets of values automatically adjust to what they were in the past? Add-only enumerations are a much simpler solution [3].

One final note: I don’t know how anyone else does design, but I do it by creating and exploring a decision tree. Each time I have a problem to solve, I list my options, along with their pros and cons. I then develop the most promising one until I hit a snag, at which point I either do a little research, write some experimental code, backtrack, or ask someone for help. “Tree” isn’t quite the right word, though; in many cases, it’s more like a partially-filled table, in which some choices can be combined with some others. Each time I explore one cell in the tree, I potentially learn more about the pros and cons of others.

Next up: all the other hard bits of tickets.

---

[1] Trac‘s tickets have, among other fields, enumerations for priority, severity, component, and version. In practice, everyone seems to fold “priority” and “severity” together in their heads: issues are either urgent, important, or ignorable. “Component” is also problematic—many issues tie into many components, while others (such as questions) don’t have anything to do with components at all—so we dropped it entirely. (Users can tag tickets with keywords to identify components if they want to.) Finally, student projects aren’t big enough, or long-lived enough, to need version labels; once again, if a project does, it can use tags.

[2] Saying that “dates are stored as dates” glosses over a medium-sized annoyance. As Victor Ng pointed out, Python’s database interface libraries will return any of several different data types to represent a moment in time, ranging from a datetime.datetime from the standard library to something as exotic as eGenix‘s mxDateTime, depending on which database and database interface you’re using.

[3] The same thing applies to user IDs. Once a user is in the system, her ID can never safely be deleted, since that would result in dangling references. Instead, DrProject stores a Boolean “enabled” flag in each account record; disabled accounts cannot be logged into, mail is not forwarded to them, etc.

DrProject, Teaching

Two of our projects students are looking for full-time jobs to start in the spring:

• Lillian Angel
• Andrey Petrov

We think highly of both of them, and we’re sure you will too

Employment