I recently received mail from someone working on a software-based approach to fault tolerance.
Their tool makes applications more reliable,
but they think it also makes developers more productive
by reducing the amount of error detection and handling code they need write.
They have never been able to find research
that quantifies how much time developers spend on code for detecting and handling problems
relative to the effort for the “happy path”.
they know it’s substantial,
and is (probably) increasing as applications become more distributed,
but the only number they’ve found is from a 1995 book called
Software Fault Tolerance,
where Dr. Flaviu Cristian says that it often accounts for more than two-thirds of code in production systems.
So I asked a dozen researchers I met through It Will Never Work in Theory
if they knew of anything,
and the answer was, “No, there isn’t anything that specifically addresses that question.”
This strikes me as odd,
because it wouldn’t be hard to measure
and the answer would be interesting.
People do throw around questionable numbers about the cost of bugs and bug fixing,
e.g., claim that companies $2 trillion in 2020.
Here are some other related resources my contacts were able to give me:
Today Was a Good Day: The Daily Life of Software Developers:
Developers spend about 11% of their time on debugging and bugfixing
with some days being dedicated to the task (up to 32%)
and some days being dedicated to meetings and collaboration (4-6%).
You can also add time spent on testing (up to 16%).
Again,
the fact that we don’t have reliable figures for this strikes me as odd.
As one of them pointed out,
while everyone is throwing LLMs at often artificial and academic problems
and then claiming to have improved some arbitrary metric X% over a random baseline,
we still don’t know fairly basic things about software development.
My thanks to everyone who responded to my late-night email about this.
For every beginning there must be an ending,
but we don’t like to talk about that,
particularly not in the tech industry.
There are thousands of books in print about how to start a business,
but only a handful about how to pass one on,
and many of those are really about how to sell out at the right time.
I have experienced a lot of endings,
and the most important thing I’ve learned is that
they can be dignified and fulfilling if done well.
I also think that preparing for the end can make it less likely,
and make what happens before it more enjoyable.
However,
a lot of people aren’t being given the chance
to wind things down gracefully.
Between the Trump administration’s attack on science
and the cuts big tech companies are making in the name of AI,
thousands of people are being given days (or less) to end years of work.
I am therefore assembling material for a half-day workshop on project closure.
If you or someone you know has ended a software project or scientific research project,
I’d be very grateful if you could spare half an hour for an online interview:
you can reach me by email at gvwilson@third-bit.com.
Note: all discussion will be confidential,
and everyone interviewed will be able to review and veto anything that mentions them
before it is seen by anyone else.
Learner Personas
There are important differences between deliberate closure
(shutting a project down of your own accord and on your own timeline),
and abrupt closure
(shutting it down on short notice under difficult circumstances).
This workshop therefore caters to two learner personas.
Vaida
Vaida, 33, has a PhD in oceanography
and now works as a data analyst for the Ministry of the Environment.
She has been collecting and publishing beach erosion data for the past six years.
She also co-founded a volunteer group that teaches environmental science to high school students,
and has been its leader for the past five years.
Vaida is relocating to pursue a new career opportunity,
so she wants to wind down her data collection project.
She also wants the volunteer group to continue its work,
but the only documentation of how it operates is
one slide deck and a couple of out-of-date blog posts.
Vaida is working hard to prepare for her new job,
which means only has two or three hours a week for the next couple of months
to put into tidying things up.
Liam
Liam, 41, worked as a civil engineer for almost a decade
before becoming a full-time software developer
at a company that does contract work
modeling slope stability for large construction projects.
While Liam writes lots of tests and uses Git and GitHub to share work with his colleagues,
very little of what he knows about OpenStabil
(the company’s open source software package)
has ever been written down.
Liam’s group was acquired by another engineering firm sixteen months ago.
After an abrupt change of leadership,
the company has decided to merge parts of OpenStabil into a closed-source tool suite
and to stop all further development of the open version.
Liam has been told to make these changes immediately;
after protest,
he has been given until the end of the week.
Liam is deeply invested in the small but tight-knit OpenStabil community,
but has a young family at home
and doesn’t dare risk being unemployed.
I was going to title this post “Two Great Tastes That Taste Great Together”,
but I expect most of my readers are too young to get the reference,
so I’ll just dive right in:
Glitch gave literally millions of people a chance to build something on the web
without having to wrestle with NPM or webpack
or set up a server
or deal with any of the other crap that Sumana Harihareswara has dubbed
inessential weirdness.
It was beautiful and useful,
but it wasn’t profitable enough for Fastly to keep it alive.
But the idea of a low-overhead in-the-browser way for the 99% to build things
didn’t start with Glitch
and hasn’t died with it either.
Projects like Webbly (source here)
are still trying to let people use the web to build the web.
However,
someone has to host these things somewhere:
who’s going to do that, and where?
More specifically,
can we construct a hosting solution that isn’t tied to a particular company
and therefore doesn’t have a singular point of failure?
Well,
what about Mastodon?
Its authors and users are deeply committed to decentralization and federation,
and more people are running servers for particular communities every day.
What if (wait, hear me out)
what if Webbly was bundled with Mastodon
so that Mastodon site admins could provide an in-the-browser page-building experience to their users
simply by saying “yes” to one configuration option?
Why would they do that?
My answer is,
“Take a look at Mastodon’s default browser interface.”
It lets you add a couple of pictures and a few links to your profile,
but that’s less than MySpace offered twenty years ago.
I am 100% certain that if Mastodon came with an easy in-browser page builder,
people would use it to create all sorts of wonderful things.
(Awful ones too, of course, but Mastodon site admins already have to grapple with content admin.)
Greenspun’s Tenth Rule is that every sufficiently complicated program
contains a mediocre implementation of Lisp.
Equally,
I think every useful web-based tool is trying to be
what Visual Basic was in the 1990s
and WordPress was to the early web:
useful, right there, and a gradual ramp for new users rather than a cliff to climb.
I think the sort of people who built useful little things with Glitch
would do amazing things with Webbly
if it was married to their social media.
I also think that allowing people to create custom home pages
or tweak their feeds
would draw a lot of new users away from fragile, centralized systems like X and Bluesky.
I know that I’ve been wrong far more often than I’ve been right,
but this really does feel promising.
Contributing to some open source project,
because I enjoy programming the way some people enjoy knitting or jigsaw puzzles.
I’d find time in there for housework,
playing the sax,
and riding my bike or swimming (depending on the season and the state of my knees).
What I really want,
though,
is a circle of friends to do these things with.
I love my wife very much,
but we have our own interests:
I’m never going to enjoy gardening or cooking as much as she does,
boardgames like Hive
or Santorini aren’t ever going to be her thing.
I always thought my younger brother and I
would wind up playing chess in the park in our twilight years;
here’s hoping I find a few other people to do that with in his absence.
Following up on my recent plea
for material to use in a workshop on productivity for software engineers,
I’d also be grateful for material on:
How to hand off or wind down a research project or open source software project.
For every beginning there must be an ending, but we don’t like to talk about them.
For example, there are thousands of books in print about how to start a business,
but only a handful about how to pass one on or wind one down,
and many of those are really about how to sell out at the right time.
I think the reason is that mortality scares us,
but preparing for the end can make the end less likely,
and can help make what happens before then more enjoyable.
If you have ever shut something down or handed it over after several years,
please get in touch.
A legal and business comparison of the new generation of software licenses.
The Business Source License
has attracted a lot of fire from open source zealots;
the Unreal Engine license
also puts restrictions on use,
but hasn’t caused nearly as much yelling.
If anyone has written a level-headed compare-and-contrast of licenses
that the Open Source Initiativehasn’t blessed,
but which give users more rights than a standard closed-source license,
I’d be very grateful for a pointer—again,
please get in touch.
Suppose you were going to build a one-day hands-on tutorial for software engineers
on what “productivity” means and how to observe it
(which might or might not include measuring it quantitatively).
You have 6x30 minutes to talk and 4x45 minutes for practical exercises.
(Yes, that’s only six hours, but you need to leave time for questions, coffee breaks, lunch, etc.)
If each of those six slots is built around one particular research paper,
what are the papers?
And what are the four exercises?
Please:
Peer-reviewed research papers please, not opinion pieces from industry pundits.
Papers, not books (but I’ll accept chapters from books as “papers”).
Tools and data used in the exercises has to be openly accessible.
If you have suggestions, please [email me][email], and thank you in advance.
If there are enough useful comments, I’ll organize an online discussion of this.
Why productivity?
Because it affords a good mix of “how do we collect and analyze data?”,
“when are quantitative and qualitative analysis the right tools?”,
and “how do we operationalize a question like this?”
It’s also very topical—most discussions of the impact of AI are vacuous
because we’re arguing about performance improvements without first defining
performance—and there’s a lot of research out there to draw on.
I also hope it will draw interest from both the research community
and research-friendly practitioners.
Update
Beller et al 2021
Moritz Beller, Vince Orgovan, Spencer Buja, and Thomas Zimmermann:
“Mind the Gap: On the Relationship Between Automatically Measured and Self-Reported Productivity”
IEEE Software,
38(5),
2021,
10.1109/MS.2020.3048200.
Czerwinski et al 2003
Mary Czerwinski, Greg Smith, Tim Regan, Brian Meyers, George G. Robertson, and Gary K. Starkweather:
“Toward Characterizing the Productivity Benefits of Very Large Displays”
Interact,
3,
2003.
Forsgren et al 2021
Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Tom Zimmermann, Brian Houck, and Jenna Butler:
“The SPACE of Developer Productivity: There’s More to It Than You Think”.
ACM Queue,
19(1),
2021,
10.1145/3454122.3454124.
Hicks et al 2024
Catherine M. Hicks, Carol S. Lee, and Morgan Ramsey:
“Developer Thriving: Four Sociocognitive Factors That Create Resilient Productivity on Software Teams”
IEEE Software,
41(4),
2024,
10.1109/MS.2024.3382957.
Meyer et al 2014
André N. Meyer, Thomas Fritz, Gail C. Murphy, and Thomas Zimmermann:
“Software Developers’ Perceptions of Productivity”
FSE‘14,
2014,
10.1145/2635868.2635892.
Murphy-Hill et al 2021
Emerson Murphy-Hill, Ciera Jaspan, Caitlin Sadowski, David Shepherd, Michael Phillips, Collin Winter, Andrea Knight, Edward Smith and Matthew Jorde:
“What Predicts Software Developers’ Productivity?”
IEEE Trans. Software Engineering,
47(3),
10.1109/TSE.2019.2900308.
Noda 2023
Abi Noda, Margaret-Anne Storey, Nicole Forsgren, and Michaela Greiler:
“DevEx: What Actually Drives Productivity”
ACM Queue,
21(4),
2023,
10.1145/3595878.
Prechelt 2019
Lutz Prechelt:
“The Mythical 10x Programmer”
In Sadowski & Thomas Zimmermann (eds.):
Rethinking Productivity in Software Engineering,
Apress,
2019,
978-1484242209.
Sadowski & Zimmermann 2019
Caitlin Sadowski and Thomas Zimmermann (eds.):
Rethinking Productivity in Software Engineering.
Apress,
2019,
978-1484242209.
Sedano et al 2017
Todd Sedano, Paul Ralph, and Cécile Péraire:
“Software Development Waste”
Proc. ICSE‘17,
2017,
10.1109/ICSE.2017.20.
Storey et al 2022
Margaret-Anne Storey, Brian Houck, and Thomas Zimmermann:
“How Developers and Managers Define and Trade Productivity for Quality”
Proc. CHASE‘22,
2022,
10.1145/3528579.3529177.
You know what would be a really thought-provoking team project for undergrad software engineering students?
Building a dashboard that measures and reports programmers’ productivity and identifies process bottlenecks.
Lots of coding,
lots of requirements gathering,
lots of UI design,
and lots of “what the hell are we actually measuring and do we believe it’s meaningful?”
which would be the most illuminating part of the whole exercise for them.
2025-08-09
My sister Sylvia
died of pancreatic cancer in 2012 at the age of 47,
leaving three teenage children behind.
The mRNA cancer vaccine being developed in the US was the best hope we’ve ever had to prevent this.
The Trump Administration’s decision to cut funding for this live-saving research based on pseudo-scientific hysteria
condemns more people like my sister to death and more children to grow up without their mothers.
2025-08-11
There is a lot of ageism in tech.
One reason,
I think,
is that 30-year-olds spouting recycled Y Combinator bullshit
don’t like having the obvious flaws and contradictions in their statements pointed out
by people who’ve lived through two or three previous hype cycles.
Not hiring the elderly ensures that they’re always in a high-adulation environment,
and never have to worry about whether staff will resist being guilted or bullied into sacrificing their personal lives
for someone else’s gain.
2025-08-12
Andreas Zeller’s work is the most admirable example of software engineering research I know of.
Over several decades he and his team have done deep theoretical work and translated it into usable tools,
both to see if their ideas work in the real world and as a basis for dialog with developers about what problems to work on next.
(Go ahead, check out The Fuzzing Book
and The Debugging Book —
you can thank me later.)
If I could go back 20 years and re-start my failed attempt to become a researcher myself,
I would try to do the same thing,
but would focus on developer productivity.
More specifically,
I would try to build deployable tools that programmers would actually use
to find bottlenecks in real-world development processes,
while simultaneously doing the theoretical work needed to figure out
“what do we actually mean by ‘productivity’ and how do we tell if someone’s productive or not?”
There are lots of products in this space (Swarmia, Flow, etc.)
but I haven’t seen something that is
(a) open source,
(b) straightforward to deploy,
(c) actively being improved,
and (d) backed up by the kind of research that Cat Hicks
and the contributors to this book do.
I think that if we had something like this today,
a lot of the debate about whether AI makes programmers more productive would have a much higher signal-to-noise ratio.
2025-08-12
If you are using AI coding tools, please copy your prompt into the generated code as a comment (or have your agent do that):
You wrote a precise requirement spec - why throw it away?
It’ll help the next person understand the code.
It’s good context for the AI when you revise the code, too.
It will help researchers who want to study these tools’ accuracy and effectiveness.
It’s a good way to label code as “AI-generated” or “AI-assisted”.
Following up on yesterday’s post,
here are some workshops I’d really like to take.
If you’re offering any of them,
please let me know.
Experimental design and analysis for software engineers
who are trying to figure out how well their AI-based tools actually work.
An introduction to the social sciences for programmers
who want to understand how power actually works in our society.
(See Sex and Drugs and Guns and Code for some thoughts.)
Distributed systems design by example—basically,
a sequel to Software Design by Example
in which learners build working models of distributed systems
in order to understand how the real things work.
Systems administration for data scientists
(or, what people who wrangle dataframes for a living need to know
in order to deploy an application for others to use).
This request is mostly professional curiosity:
I’ve tried and failed to create this,
and would like to see how others do it.
Managing research software projects—not “how to use Git”
but hiring and firing,
marketing,
budgeting,
succession planning,
and similar challenges.
Again,
I’ve also tried and failed to create this,
so my interest here is partly professional curiosity.
Organizational change for research software engineers.
We spend a lot of our time rolling rocks up hills;
as I approach the end of my career,
I’d really like someone to teach me how to get rid of the damn hills.
Note: we’re working on this one;
please reach out if you’d like to help.
In a better world than this,
Canada would have a Natural Science and Engineering Education Council alongside NSERC
to fund the creation, dissemination, delivery, and evaluation of teaching and training materials.
Alas, that is not the timeline we’re in…