Somewhere Between Hobbes and Wishful Thinking


A couple of days ago the news broke that a company called Koko1 used ChatGPT to provide therapy online to thousands of people without telling them and without getting any sort of ethics clearance, then claimed that this was somehow magically exempt from informed consent law. I didn’t have anything to say about this that better-informed people hadn’t already said, but it got me wondering yet again: what can we teach undergrad software engineers that will convince them that doing things like this is wrong? We can tell them, but my experience is that if they don’t already believe, they will tell us what they know we want to hear on the exam but nothing in their behavior will change2.

When I posted that thought on Mastodon most of the replies fell into one of two categories:

I’m grateful for the feedback, but I disagree with both of these positions. I agree with the person who pointed out that fraud and unscrupulous profiteering crop up whenever safeguards in the financial industry are relaxed, but I don’t think fear of legal penalty is the only (or even main) reason that medical professionals protect patients’ privacy or warn them about side effects. I could be fooling myself, but empirically, Hobbes was wrong: as Rebecca Solnit and many others have pointed out, most people work together most of the time rather than turning on each other.

At the other end of the spectrum, saying that we need to completely overhaul our schools is like saying that we need to stop racism, end climate change, or dismantle capitalism. These statements all beg the question, “What do I do today with the resources I have?” If I ever teach an undergrad software engineering class again, I will have roughly thirty hours in which to lecture a room full of twenty-year olds; they will then have to spent about sixty hours doing assignments and then write a final exam. That’s it: that’s what I have to work with, so telling me there’s a better world over there somewhere3 doesn’t help.

I agree that we need legal penalties for misbehavior and legal protection for whistleblowers and disclosure requirements and everything else, but I believe that unless people have a moral compass, those rules and regulations will have no effect. I also believe that morality is teachable because I’ve seen it. My father was a good man, but by today’s standards he was mildly racist and moderately homophobic. I believe my daughter and her friends are much less of either—not just because they’re afraid of penalties, but because we taught them to be better. A single undergrad class when someone is twenty years old (plus or minus) might not have much impact, but I’d still like to try.

  1. But really, a guy named Robert Morris, because companies don’t make decisions—people do.

  2. I’ve written three quarters of an undergrad introduction to software engineering twice in the last six years and set it aside without finishing it each time because this problem is a lot more important than design patterns or dev ops If I can’t figure out how to have some impact here, it seems kind of pointless to rearrange the digital deck chairs.

  3. /me waves hand vaguely at the window.