NOAA’s Dealing with Disruptive Behaviors
describes ten different kinds of people who can disrupt meetings and gives advice for handling each.
I’ve been a big fan of this taxonomy since I first encountered it several years ago,
but going through it again yesterday,
I realized that there’s a fair bit of overlap between some of the types.
In order to slim it down,
I grouped them as follows:
Aggressive
Assertive
Passive
People-focused
Blowfish, Dolphin
Sea Otter, Flounder
Task-focused
Jellyfish, Shark
Sea Lion
Crab, Clam, Octopus
I then picked one from each group,
looked at the result,
and added Clam back into the mix
because I felt this type was distinct enough from Crab to merit inclusion.
This came up as part of an exercise to analyze
how people are responding to AI mandates (both positively and negatively)
and how to manage those responses.
If you have seen other work that looks at this,
or have feedback on how I’ve slimmed these categories down,
I’d be grateful for feedback.
Talkative Blowfish
Blowfish is a very chatty, assertive people person.
Blowfish wants everyone to feel comfortable and positive about the process.
They have a tendency to be overly talkative (almost compulsively)
because they are enthusiastic, want to show off, or are well-informed and eager to use their knowledge.
Blowfish can dominate the floor time at the expense of other group members.
While they frequently have good ideas and strong contributions to make,
they also ramble, monopolize the discussion, and do not give others an opportunity to express their thoughts.
Subsumes Dolphin: Blowfish rambles about the work, Dolphin diverts from it.
Eager Sea Otter
The sea otter is a passive people person and wants everyone to get along.
Sea otters are super agreeable, overly positive people
who are optimistic, very reasonable, sincere, and supportive.
They are people-oriented and aim to please those nearby (for instance, by always saying yes).
They seek approval by giving approval.
This may cause difficulty in group situations because they overcommit or are unreliable.
Subsumes Flounder: the Sea Otter is eager to please, not detached.
Dominating Shark
The shark is aggressive and focused on efficiency and the task.
They can be hostile and dominating and might try to intimidate and bully people.
They make cutting remarks or throw temper tantrums when they do not get their own way.
Some hostile individuals will be task-focused and want to get the job done while maintaining control.
These individuals will generally have a more focused attack
on the failure of others to complete a specific task or take necessary actions.
Others may explode and attack other people in a more random fashion,
which is typically done to command attention.
Subsumes Jellyfish: the Shark’s aggression is about domination, not intellectual sport.
Complaining Crab
The complainer can come in many forms: whiner, critic, or obstructionist.
Crabs are passive and task-focused, and they want to get it done.
Despite the negative connotation, this person is often motivated by perfection.
Negative, complaining people may seem to object to everything,
asserting that ideas proposed will not work or are impossible.
The complainer may completely deflate any optimism others express for a project
and may block others from accomplishing goals.
Crabs gripe and do little to improve the situation,
either because they feel powerless
or because they refuse to bear the responsibility for an imperfect solution.
Subsumes Octopus: the Crab complains outward rather than freezing inward.
Arrogant Sea Lion
Sea lions are assertive and need the group to accept their expertise,
and they can become know-it-alls when questioned.
They believe that they have more credibility than has been acknowledged
and want everyone to understand and agree with them.
The sea lion knows a lot about the topic but does not contribute in a way that sits well with other participants,
sometimes using their credentials, age, length of service, or residency to disparage an idea.
With cockiness and an inflated ego,
the sea lion can be condescending, imposing, pompous, or arrogant toward others.
In all likelihood, this behavior will make others feel as though there is no point in contributing.
Shy Clam
The clam is shy and quiet, passive, and task-focused.
Clam wants to get it right.
Shy individuals may be reluctant or afraid to express their ideas in a group setting,
so they may appear to be unresponsive.
I’m co-teaching a lesson for the Carpentries next week
about the impact of LLMs on teaching.
Here are a few things I’ve been reading to prepare:
Barba2026
Lorena A. Barba and Laura Stegner:
“The Conversational Exam: A Scalable Assessment Design for the AI Era”.
https://arxiv.org/abs/2601.10691,
2026.
Conversational exam (live coding + explanation in small groups) restores assessment validity against generative AI cheating; 58 students examined in 2 days; combines authentic practice with inherent validity.
Bielaczyc1995
Katerine Bielaczyc, Peter L. Pirolli, and Ann L. Brown:
“Training in Self-Explanation and Self-Regulation Strategies: Investigating the Effects of Knowledge Acquisition Activities on Problem Solving.”
Cognition and Instruction.
13(6),
1995.
https://doi.org/10.1207/s1532690xci1302_3.
Training study (24 novice programmers) showing self-explanation and self-regulation strategy training causally improves programming task performance; instructional group showed significantly greater strategy use and performance gains.
Bridgeford2025
Eric W. Bridgeford, Iain Campbell, Zijao Chen, et al.:
Ten Simple Rules for AI-Assisted Coding in Science.
https://arxiv.org/abs/2510.22254,
2025.
10 practical rules for AI-assisted coding in scientific computing; addresses problem preparation, context management, testing/validation, and code quality; emphasizes human agency and domain expertise for reproducible research.
Butler2024
Jenna Butler, Jina Suh, Sankeerti Haniyur, and Constance Hadley:
“Dear Diary: A Randomized Controlled Trial of Generative AI Coding Tools in the Workplace.”
https://doi.org/10.48550/arxiv.2410.18334,
2024.
Mixed-methods study (survey + RCT + 3-week diary) on generative AI coding tools at a large multinational; sustained use increases perceived usefulness and enjoyment; trustworthiness perceptions unchanged; 84% report positive daily work changes; unexpected uses include web search replacement.
Deslauriers2019
Louis Deslauriers, Logan S. McCarty, Kelly Miller, Kristina Callaghan, and Greg Kestin:
“Measuring Actual Learning Versus Feeling of Learning in Response to Being Actively Engaged in the Classroom.”
Proc. National Academy of Sciences,
116,
Sept. 2019.
https://doi.org/10.1073/pnas.1821936116.
RCT shows active learning produces more learning but lower perceived learning; increased cognitive effort is misread as poorer learning; early instructor intervention corrects this misperception.
FinnieAnsley2022
James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather:
“The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming.”
Proc. 24th Australasian Computing Education Conference,
https://doi.org/10.1145/3511861.3511863,
2022.
OpenAI Codex outscores most students on intro programming exams; handles Rainfall problem variants well; generates diverse solutions for identical prompts; raises challenges and opportunities for CS education.
Jiao2026
Yuling Jiao and Qiuli Wang:
“Large language models for formative feedback in writing instruction: a systematic review of classroom interventions, feedback quality, and student outcomes”.
Frontiers in Education,
11,
2026,
https://doi.org/10.3389/feduc.2026.1834085.
Studies in which teachers discussed AI-generated feedback, helped students interpret it, or combined it with their own comments generally reported better learning outcomes than studies where students worked independently with AI.
Leinonen2023a
Juho Leinonen, Paul Denny, Stephen MacNeil, et al.:
“Comparing Code Explanations Created by Students and Large Language Models.”
Proc. 2023 Conference on Innovation and Technology in Computer Science Education,
https://doi.org/10.1145/3587102.3588785,
2023.
LLM-generated code explanations are rated significantly more accurate and understandable than student-generated ones in a 1000-student course; scalable on-demand explanations can scaffold introductory programming learning.
Leinonen2023b
Juho Leinonen, Arto Hellas, Sami Sarsa, et al.:
“Using Large Language Models to Enhance Programming Error Messages.”
Proc. 54th ACM Technical Symposium on Computer Science Education,
https://doi.org/10.1145/3545945.3569770,
2023.
LLMs enhance Python error messages with plain-language explanations and fix suggestions; sometimes surpass original messages in interpretability and actionability for novice programmers.
Ma2024
Qianou Ma, Hua Shen, Kenneth Koedinger, and Sherry Tongshuang Wu:
“How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debugging.”
Lecture Notes in Computer Science,
https://doi.org/10.1007/978-3-031-64302-6_19,
2024.
HypoCompass trains students to debug LLM code by having them hypothesize error causes while LLMs handle code completion; improves debugging performance 12% over pre-test with fourfold efficiency vs. human tutors.
Ma2025
Qianou Ma, Weirui Peng, Chenyang Yang, Hua Shen, Ken Koedinger, and Tongshuang Wu:
“What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use.”
ACM Transactions on Computer-Human Interaction,
32(4),
https://doi.org/10.1145/3731756,
2025.
Randomized experiment with 30 novices finds Requirement-Oriented Prompt Engineering (ROPE) training achieves 20% gains vs. 1% for conventional prompt engineering training.
OBrien2026
Gabrielle O’Brien, Alexis Parker, Nasir Eisty, and Jeffrey Carver:
“A survey of generative AI adoption and perceived productivity among scientists who program.”
2026,
https://doi.org/10.48550/arXiv.2512.19644.
Survey of 868 scientists who program as part of their work,
reporting that 80% use GenAI tools in their programming,
with 77.5% of those using general purposing tools like ChatGPT over specialised coding tools.
Peng2023
Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer:
“The Impact of AI on Developer Productivity: Evidence from GitHub Copilot.”
2023,
https://doi.org/10.48550/arXiv.2302.06590.
Randomized controlled experiment claiming that GitHub Copilot
users completed a JavaScript coding task 55.8% faster than the
control group.
Richards2026
Jonan Richards, Bruno Alves de Oliveira, Iury Oliveira, Igor Wiese, and Mairieli Wessel:
“No Two Developers Think Alike: How Problem-Solving Styles and Experience Shape Needs in Conversational Interaction with Copilot”.
2026,
https://arxiv.org/abs/2606.19216.
Characterizes 5 distinct interaction modes and 10 underlying needs in developers’ interactions with AI tools.
Sadowski and Zimmerman 2019
Caitlin Sadowski and Thomas Zimmermann (eds.):
Rethinking Productivity in Software Engineering.
Apress,
2019,
9781484242216.
Edited volume collecting research and practitioner perspectives on
how to understand, define, and measure software developer
productivity.
Stray2026
Viktoria Stray, Elias Goldmann Brandtzæg, Viggo Wivestad, Astri Barbala, and Nils Brede Moe:
“Developer Productivity with and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study.”
Proceedings of the 59th Hawaii International Conference on System Sciences,
https://doi.org/10.24251/hicss.2026.880.
2026.
Mixed-methods study of 703 NAV IT repositories finds Copilot users were more active even before adoption and shows no statistically significant changes in commit-based metrics after adopting the tool.
I finally have time to flesh out ideas for lessons that I’ve wanted for years.
However,
if I can’t find a way to send them back to 2006,
there’s no point writing them:
very few people read long-form tutorials about software these days.
I’d still be interested in comments, though—figuring out what I would teach
always helps me learn.
Overview
Topic: Error handling.
Audience: Senior undergraduates who are
comfortable writing programs in Python and JavaScript that are hundred of lines long,
know how to raise and catch exceptions,
and are familiar with SQL, C, and the Unix shell,
but have no experience building production-robust programs.
Format: seven 45-minute lessons with exercises.
1) What Can Go Wrong
Taxonomy of errors
Logic errors: bugs in the program itself
Runtime errors: null dereference, division by zero, index out of bounds
External errors: file not found, network timeout, database constraint violation
Human errors: bad input, misconfiguration, wrong file format
Environmental errors: disk full, out of memory, clock skew
Failure modes
Fail-fast vs. fail-slow: silent corruption is harder to debug than an early crash
Silent failures: errors ignored, wrong results returned without warning
Cascading failures: one component’s error triggering failures in others
C-style error signaling
Return codes: functions return -1, NULL, or 0 on failure
errno: a global (thread-local) integer set by system calls
Read with perror() or strerror()
Sentinel values: EOF (-1), invalid index, or a special out-of-band value
Advantages: explicit control flow, no hidden jumps, zero runtime overhead
Disadvantages: easy to ignore, verbose, callers must check every call, no stack information
Common pitfalls:
Not checking return values
Reading errno after another call has overwritten it
errno not being set on success
Philosophy: errors are not exceptional, they are expected
The happy path is one of many paths
Spectrum of responses: ignore vs. crash vs. recover vs. degrade gracefully
Choosing a response requires knowing the context and the cost of each option
Taxonomy drill
Given six short programs in Python, JavaScript, and C,
each containing a different type of error,
classify each error using the taxonomy above
and explain whether the program’s response (crash, wrong output, hang, silent skip)
is appropriate for a production context.
errno pitfall hunt
A short C function uses fopen, fread, and fclose and checks errno after each call.
The function contains three bugs related to C-style error handling
(e.g., ignoring a return value, checking errno too late, not distinguishing error from end-of-file).
Identify each bug and propose a fix.
Failure brainstorm
Given a brief description of a web form that accepts a user’s name, email, and a file upload,
then stores the data in a database,
list every error that could occur,
including errors the user causes,
errors the network causes,
errors the OS causes,
and errors the program itself could cause.
Compare lists with a partner and identify any category you missed.
2) Error Propagation and Recovery
Options when an error is detected
Crash/abort: call abort(), panic, or let the process die
Appropriate when the program is in an unrecoverable state or an invariant is violated
Raise an exception: hands control to the caller’s handler
Log and continue: almost always wrong
Hides failures and corrupts program state
Retry: attempt the operation again (only safe for transient, idempotent operations)
Use a fallback value: return a default, a cached result, or a degraded response
Partial success: complete what you can, report what failed
Compensating action: undo work already done before propagating the error
Propagating errors without losing information
Re-raise: pass the error up unchanged
Wrap/chain: add context while preserving the original cause (Python raise X from Y, Java initCause)
Translate: convert a low-level error into a domain-level error
(e.g., from OSError to ConfigurationError)
Anti-pattern: catching an exception and throwing a new one that discards the original
Adding context as errors propagate
Include what was being attempted and with what inputs (sanitized)
Each layer adds the context it has
Callers should not have to guess
C-style propagation patterns
Check every return value, every time: a missed check is a latent bug
Passing error information up via out-parameters or a shared error struct
goto cleanup pattern for resource cleanup on multiple error paths
Comparing C-style propagation to exception propagation: explicitness vs. verbosity
Retry logic
Distinguish transient errors (timeout, temporary unavailability)
from permanent errors (permission denied, not found)
Exponential backoff: double the wait time between retries
Add jitter (random variation) to prevent thundering herd
Set a maximum number of retries and a total timeout
Only retry idempotent operations
Retrying a non-idempotent operation can cause duplicate side effects
Propagation rewrite
A Python function reads a configuration file,
connects to a database using values from that file,
and runs a query.
The function is given with no error handling.
Rewrite it so that errors at each stage are wrapped with context and propagated appropriately,
then rewrite the equivalent in C using return codes, noting what is harder and easier in each style.
Swallowed exceptions
A code snippet contains three except: pass or equivalent constructs.
For each, explain what failure is being hidden,
what could go wrong as a result,
and what the correct handling would be.
At least one case should be a place where logging-and-continuing is wrong even though it feels safe.
Retry with backoff
Implement a retry decorator (Python) or higher-order function (JavaScript) that wraps a function call,
retries up to N times on specified exception types,
uses exponential backoff with jitter,
and raises the last exception if all retries fail.
Test it against a stub that fails a configurable number of times before succeeding.
Discuss which HTTP status codes should trigger a retry and which should not.
3) Error Communication
Errors have multiple audiences with different needs
End users: need to know what happened and what they can do about it
API callers: need structured, machine-readable information to handle programmatically
Operators and on-call engineers: need enough detail to diagnose and fix the problem
User-facing error messages
State what went wrong in plain language
Tell the user what to do next (retry, contact support, correct their input)
Avoid technical jargon, stack traces, and internal identifiers
Do not blame the user
Provide a reference (request ID, error code) they can give to support without revealing internals
Distinguish “you did something wrong” (4xx) from “we did something wrong” (5xx)
API error responses
HTTP status codes: 4xx for client errors, 5xx for server errors
Use specific codes (400, 401, 403, 404, 409, 422, 429, 503) rather than always returning 400 or 500
Error response body: include type, message, detail, and request_id fields
RFC 7807 (Problem Details for HTTP APIs): a standard format worth knowing
Consistent schema across all endpoints
Callers should not have to handle different shapes
Validation errors: return all field errors at once, not just the first one
Security considerations in error communication
Information leakage:
stack traces, SQL queries, file paths, and internal service names in error messages help attackers
User enumeration: “user not found” vs. “wrong password” reveals whether an account exists
Use “invalid credentials” instead
Timing side-channels: if “user not found” returns in 1ms and “wrong password” returns in 200ms
(due to password hashing),
an attacker can enumerate accounts by timing
Always perform the same operations regardless of the error branch
Error messages as reconnaissance:
the more specific the error, the more an attacker learns about your system
Rule of thumb:
the user-facing message and the internal log entry should contain different levels of detail
Correlation IDs
Generate a unique ID per request
Include it in all logs and in the user-visible error response
Allows an operator to find all logs related to a specific error report
Pass the ID through every internal service call in a distributed system
Message rewrite
Five error messages from real or realistic applications are provided
(e.g., a raw Python traceback shown in a web UI,
a message reading “MySQL error 1045 for user ‘admin’@’localhost’”,
a message reading “Your session token has expired and cannot be refreshed”).
For each,
write a user-appropriate replacement that is informative without leaking internals,
and write a separate message suitable for the internal log.
API error schema design
Design the error response body for a REST endpoint that registers a new user.
The schema must handle missing required fields,
fields that fail validation (email format, password length),
a duplicate username,
and an unexpected server failure.
Show example JSON for each case.
Login timing attack
A login function is provided that queries the database first
and returns “User not found” immediately if the user does not exist,
or hashes the password and compares it (taking ~200ms) if the user does exist.
Explain the timing side-channel,
fix it so both branches take the same time,
and discuss whether this fix is always necessary or whether it depends on the threat model.
4) Logging and Observability
Why logging matters for error handling
Errors that are caught and handled still need to be visible to operators
Post-mortem debugging requires a record of what happened
Trends in error rates reveal systemic problems before they become outages
Log levels and when to use each
DEBUG: detailed internal state, useful during development, not in production by default
INFO: normal significant events (startup, shutdown, completed request)
WARNING: unexpected but handled conditions that may indicate a problem
ERROR: a failure that was caught but means something did not complete successfully
CRITICAL: system is in a severely degraded or unrecoverable state
Common mistake: logging every caught exception at ERROR regardless of severity
Structured logging
Free-text logs are hard to query: key-value pairs (or JSON) are machine-parseable
Standard fields: timestamp, level, message, request_id, user_id, error_type, duration_ms
Log the event, not a sentence:
{"event": "db_query_failed", "table": "users", "error": "timeout"}
rather than "Failed to query users table due to timeout"
What to include in an error log entry
What was being attempted and with what parameters (sanitized)
The error: type, message, and stack trace for unexpected errors (not for expected/handled errors)
The outcome: did the system recover, degrade, or fail?
Correlation ID to link this entry to the request and to other services
What NOT to log
Passwords, API keys, session tokens, OAuth codes: not even partial values or hashes
Personally identifiable information (PII): names, emails, government IDs
Check compliance requirements (GDPR, HIPAA)
Full request/response bodies if they may contain credentials or sensitive data
Stack traces in user-facing API responses (they belong in the internal log only)
Audit logs vs. diagnostic logs
Diagnostic logs: help engineers debug problems (mutable, short retention, lower protection)
Audit logs: immutable record of security-relevant actions
(who authenticated, what data was accessed, what was changed)
Long retention, tamper-evident, access-controlled
Not the same stream: conflating them causes both compliance and debugging problems
Traces: end-to-end path of a request through multiple services
Correlation IDs connect all three (errors visible in metrics can be traced to specific log entries)
Logging retrofit
A function that processes uploaded CSV files
is given with a single except Exception as e: print(e) handler.
Add structured logging using Python’s logging module (or a JavaScript equivalent).
Include appropriate log levels for different error conditions,
add relevant context fields,
distinguish expected errors (bad CSV format) from unexpected errors (disk full),
and identify what was unobservable before your changes.
Security audit of log output
Three log excerpts are provided,
each containing at least one security violation
(e.g., a plaintext password,
a full stack trace that reveals an internal file path and SQL query,
or a user ID that allows enumeration of account existence).
Identify each violation, explain what risk it creates, and show the corrected log entry.
Logging strategy design
A data pipeline reads records from an API, transforms them, and writes results to a database.
The pipeline processes records in batches.
Design a logging strategy: what events are logged at each stage,
what fields each entry includes,
what constitutes a loggable error vs. a metric,
and how an operator would diagnose a job that completed without crashing
but produced fewer output records than expected.
5) External Systems
Why external systems are the primary source of production errors
Code you write can be tested exhaustively
Systems you depend on cannot be controlled
External systems fail in ways that are partial, delayed, and inconsistent
File system errors
Not found, permission denied, disk full, file locked by another process, corrupted data
Always close files: use context managers (with in Python, using in C#, RAII in C++)
Atomic writes: write to a temporary file, then rename
Rename is atomic on POSIX systems
Direct writes leave a window where the file is partially written
Handling partial reads and partial writes explicitly
Database errors
Connection failure: the database is unreachable (transient, so retry with backoff)
Timeout: query took too long (may or may not have committed, so check before retrying)
Constraint violation: duplicate key, foreign key failure, not-null violation (permanent, so do not retry)
Deadlock: two transactions are waiting on each other (transient, so retry the whole transaction)
Serialization failure (in serializable isolation):
transaction conflicted with a concurrent one (transient, so retry)
Connection pool exhaustion: all connections are in use (application-level backpressure needed)
Distinguishing transient from permanent errors using error codes, not message strings
Network and HTTP errors
Every external call must have a timeout: separate connect timeout and read/total timeout
Retrying safely:
only retry if the operation is idempotent or the error occurred before the request was received
HTTP status codes that indicate retrying is safe: 429 (with Retry-After), 503, 504
HTTP status codes that should not be retried: 400, 401, 403, 404, 409, 422
Exponential backoff with jitter (review from Lesson 2, now applied to real HTTP clients)
Never assume the schema of data from an external source
Validate presence and type of required fields before using them
Handle missing optional fields explicitly rather than letting a KeyError or undefined propagate
Version skew: the API changed its schema (your parser must detect and report this)
Fail loudly on unexpected schema changes rather than silently producing wrong results
Error handling retrofit for a pipeline
A script that downloads a JSON file from an HTTP endpoint,
parses it,
and inserts records into a SQLite database is given.
It has no error handling.
Add appropriate handling for
HTTP errors (including distinguishing retryable from non-retryable),
JSON parse failures,
missing required fields,
and database constraint violations.
For each error type, decide whether to abort, skip the record, or retry, and justify the choice.
Transaction retry
A function executes a multi-statement database transaction and fails with a deadlock error.
Implement retry logic that retries the entire transaction on deadlock,
does not retry on constraint violations,
limits total retries,
and logs each retry attempt.
Discuss why retrying only the failed statement, rather than the whole transaction, is wrong.
Atomic file write
A function saves user settings to a JSON file by opening the file,
serializing the settings,
and writing directly.
Demonstrate two failure scenarios where this approach corrupts the file:
interrupted write,
and crash between open and close.
Implement the temp-file-then-rename pattern and explain why it is safe
even if the process is killed mid-write.
6) Concurrency and Async Errors
Why concurrency changes error handling
An error in a thread or task may not propagate to the code that started it
Operations may be partially complete when an error occurs, leaving shared state inconsistent
Some errors (race conditions, deadlocks) only appear under concurrency and are hard to reproduce
Errors in threads
Python: an uncaught exception in a Thread prints a traceback
but does not crash the main thread or propagate to the caller
Errors are silently lost unless explicitly collected
Thread pools (concurrent.futures.ThreadPoolExecutor):
exceptions are stored and re-raised when future.result() is called
But only if you call it
Always collect results from thread pools
Never fire-and-forget threads that could fail silently
Setting a global thread exception handler (threading.excepthook) as a backstop
Async/await errors (Python and JavaScript)
Python: an unawaited coroutine silently does nothing (forgetting await is a silent bug)
Python: an unhandled exception in a Task is reported when the task is garbage-collected
Too late to be useful
Always await tasks or attach .add_done_callback
JavaScript: an unhandled promise rejection crashes Node.js in recent versions
In older versions (and some browsers) it is silently swallowed
Always handle rejections: .catch(), try/await/catch, or process.on('unhandledRejection')
Forgetting await in a loop:
tasks are created but not awaited, the loop exits, and all tasks are cancelled
Parallel operations and partial failure
Promise.all: fails fast
If any promise rejects, the whole thing rejects and other results are lost
Promise.allSettled: waits for all promises regardless of outcome
Returns an array of {status, value/reason}
Use when you want partial results
Python asyncio.gather(return_exceptions=True):
returns exceptions as values instead of raising them
Allows processing partial results
Decision: whether to fail fast or collect all results depends on whether partial results are useful
Structured concurrency
The problem: a task that spawns subtasks and then throws has left orphaned subtasks running
Python asyncio.TaskGroup (3.11+):
all tasks in the group are cancelled if any raises an exception
The group does not exit until all tasks are done
Structured concurrency ensures the lifetime of every subtask is bounded by the scope that created it
Cancellation propagation:
when a task is cancelled, it receives a CancelledError (Python) or AbortError (JS)
Cleanup must happen in finally or try/catch around await
Shared state and locks in error paths
A lock must always be released, even if an error occurs
Use try/finally or a context manager
If an error occurs while holding a lock after modifying shared state,
the state may be inconsistent when the lock is released
Decide if the modification needs to be rolled back before releasing
Deadlock: two threads each hold a lock the other needs
Prevent via lock ordering (always acquire locks in the same order) or via timeouts
Silent thread failures
A Python script spawns ten threads,
each of which downloads a file and writes it to disk.
When a download fails,
the exception is printed to stderr but the main thread sees all tasks as complete.
Rewrite using ThreadPoolExecutor,
collect all futures,
and report which downloads succeeded and which failed,
then introduce a deliberate failure in two of the threads
and verify the errors are caught and reported correctly.
Promise.all to Promise.allSettled
A JavaScript function fetches data from five independent APIs using Promise.all.
The whole function fails if any one API is unavailable.
Rewrite it using Promise.allSettled so that
results from available APIs are returned and failures are reported per-API.
When is Promise.all’s fail-fast behavior preferable?
Deadlock identification and fix
A function managing a shared cache acquires cache_lock and then stats_lock in that order.
Another function acquires the same locks in the opposite order.
Trace through a scenario where both functions run concurrently and deadlock.
Fix the bug using lock ordering.
As a second fix,
add a timeout to the lock acquisition and show how to handle the case where the timeout expires.
7) Resilience Patterns and Production Practices
The goal: systems that degrade gracefully rather than fail catastrophically
“Fail safe” vs. “fail secure” vs. “fail operational”
No system achieves zero errors: the goal is bounded, predictable failure
Circuit breaker pattern
Closed state: requests pass through normally
Open state: requests fail immediately without attempting the operation
(protecting a downstream that is already failing)
Half-open state: a probe request is allowed through
If it succeeds, the breaker closes
If not, it stays open
Thresholds: open after N failures in a time window
Reset after a timeout
Use case:
preventing cascading failures when a slow or failing dependency causes your thread pool to fill up
Bulkhead pattern
Isolate resources (thread pools, connection pools, processes)
so that a failure in one area does not exhaust resources for another
Example: use separate HTTP connection pools for critical and non-critical external services
Trade-off: resource isolation requires over-provisioning in aggregate
Timeout patterns
Every call to an external system must have a timeout
A call without a timeout can block forever
Cascading timeouts:
internal timeouts should be shorter than the external deadline so the caller still gets a response
Distinguishing timeout-on-connect from timeout-on-read
Graceful degradation
Serve stale cached data rather than failing when the data source is unavailable
Disable non-critical features (recommendations, analytics) when load is high or dependencies are down
Return partial results rather than failing when some data is unavailable
Read-only mode: allow reads but reject writes when the system cannot safely persist data
Testing for failure
Happy-path tests verify normal behavior
Error-path tests verify that errors are handled correctly
Chaos engineering: inject controlled failures in production
to find weaknesses before an actual incident does
Error budgets and SLOs
Service Level Indicator (SLI): a measurement of reliability (e.g., fraction of requests that succeed)
Service Level Objective (SLO): the target (e.g., 99.9% success over 30 days)
Error budget: the allowed failure rate (e.g., 0.1% of requests, or about 43 minutes of downtime per month)
If the error budget is exhausted, new features stop and reliability work takes priority
Post-mortems
Document what happened, what the impact was, and why it happened
Blameless culture: focus on systemic causes, not individual mistakes
Five Whys: ask “why” repeatedly to find the root cause rather than the proximate cause
Action items must be concrete, assigned, and time-bound
Circuit breaker implementation
Implement a CircuitBreaker class in Python or JavaScript that wraps a function.
It should track consecutive failures,
open after a configurable threshold,
fail fast while open,
and attempt recovery after a timeout.
Write tests that simulate a dependency that fails, recovers, and fails again,
verifying the breaker transitions through all three states correctly.
Single points of failure analysis
A diagram shows a web application with a load balancer,
two application servers,
one database,
and one external payment API.
Each component can fail independently.
Identify the single points of failure,
propose bulkhead and timeout strategies to limit the blast radius of each failure,
and discuss the cost (infrastructure, complexity) of each mitigation.
At what point does additional resilience add more complexity than it is worth?
Error path test coverage
A small data processing function is provided,
along with a test suite with 100% line coverage on the happy path.
Enumerate all error paths in the function
(e.g., invalid input, missing file, network failure, and malformed response).
Write tests for each error path using fault injection (i.e., mock the failing component).
Measure what fraction of error paths were untested before,
and explain why line coverage is a misleading metric for error-handling code.
Appendix: Exceptions
Exception hierarchies
Python: BaseException vs. Exception vs. specific types
KeyboardInterrupt and SystemExit inherit from BaseException, not Exception
JavaScript: Error base class plus built-in subtypes (TypeError, RangeError, SyntaxError, etc.)
Catching a parent class catches all subclasses
except Exception in Python does not catch KeyboardInterrupt
Custom exception types
Subclass the appropriate base: add fields for structured error data
Use custom exceptions to distinguish your errors from library errors
and to allow callers to catch specifically
Name exceptions as nouns describing the condition,
not the action (ConfigurationError, not FailedToLoadConfig)
Exception chaining
Python: raise NewException("context") from original_exception preserves the original traceback
Python: raise NewException("context") inside an except block implicitly chains
(visible as “During handling of the above exception, another exception occurred”)
Java: pass the original exception to the constructor of the new one (new RuntimeException("msg", cause))
Cleanup with finally and context managers
finally runs whether or not an exception was raised
Use for resource cleanup (closing files, releasing locks)
Context managers (with statement) encapsulate the try/finally pattern
Prefer them for resource management
__exit__ receives exception information and can suppress the exception by returning True
Do this rarely and intentionally
Anti-patterns
except Exception: pass: silently discards all errors
except Exception as e: print(e): visible but unactionable
Provides no context and does not propagate
Catching overly broad types: bare except: in Python catches KeyboardInterrupt and SystemExit
Raising Exception directly instead of a specific type: callers cannot catch it selectively
Checked vs. unchecked exceptions (Java)
Checked exceptions must be declared or caught
Unchecked (RuntimeException subclasses) need not be
Checked exceptions enforce handling at compile time
but lead to verbose, often-ignored throws declarations
Most modern languages (Python, C#, Kotlin, Swift) do not have checked exceptions
Would you like to have some real impact on the tech industry?
Do you have $100,000 to spend?
If you answered “yes” to both questions,
ask software engineering researchers
(the kinds of people who participated in It Will Never Work in Theory)
to design a study that companies could run internally
to measure the impact that genAI adoption by programmers is having on business outcomes.
Spend $50K to get expert reviews from both practitioners and (other) researchers,
publish all of the proposals with the reviews,
and award prizes of $25K, $15K, and $10K to the three best proposals.
(If you really want to have an impact,
do this in two rounds so that participants can hybridize their best ideas.)
$100K feels like a lot of money…
Really? Compared to what you’re spending on tokens?
Does anybody actually know how to measure genAI’s impact?
It’ll be interesting to find out.
(After all, “yes” and “no” are equally interesting answers.)
Will people write decent proposals for just a few thousand dollars?
No, but they’ll do it for the attention,
and for the chance to be involved in running the study if their proposal is a winner.
I’m currently making a few last changes to the third book in this series and trying to find an agent who will handle them. If you have middle-graders who would be interested in reading them and giving me feedback, please give me a shout.
Maddy Roo
Maddy Roo takes place in a world of anthropomorphic animals and patchwork robots. Its protagonist, Maddy, is a 12-year-old kangaroo whose younger sister, Sindy, is a “throwback” with no fur, scales, or tail. Their father was kidnapped by a raiding band of robots two years before the story opens; they and their mother have struggled to make ends meet since then.
While Maddy is out one evening with a goat boy named Gumption they rescue a damaged robot from a stream. Its regulator has been broken, which allows it to reveal that another raid is about to take place. Maddy and Gumption rush back to town to warn everyone. The raiders are driven off, but not before taking Maddy’s sister and two other children.
Maddy teams up with the rescued robot, Dockety, to get her sister back. The pair manage to catch up with the raiders and free the prisoners, but in the confusion that follows, Maddy, Sindy, and Dockety are stranded in a dangerous swamp called the Mire. They take refuge in an abandoned bunker, only to discover that it is the lair of a mad robot named Patient in Darkness, who is responsible for the raiding parties.
The trio escapes by bolting a flying suit onto Dockety and get back to town moments ahead of the raid. In the aftermath of the battle that follows, Maddy realizes that she knows how to free the bots that Patient has enslaved. She uses the flying suit to return to the bunker and break Patient’s control. As the story ends, Dockety reveals that Maddy’s father is still alive and is being held prisoner in the bot city of Heck.
In Heck
In Heck picks up several months after Maddy Roo. Dockety’s community of free bots has settled just outside Rusty Bridge, and Dockety has confirmed that Maddy’s father being held in the bot city of Heck. When Sindy accidentally activates a visiting Operator’s tech during a school demonstration and badly burns Special Leaf, the Operators insist on taking her to their headquarters in Sandy Bend, even though Special Leaf warns Maddy not to let them.
Maddy and Gumption stow away in the Operators’ wagon, but a rogue flying bot kidnaps Maddy mid-journey and delivers her to the mad bot Patient in Darkness. Patient claims the Operators in league with Central (the AI that controls Heck) which plans to exploit Sindy’s ability to activate Maker technology. Maddy escapes with a discombobulator that hides her from machines and a small cleaning bot she names Mouse, makes her way to Heck, and watches helplessly as the Operators hand Sindy over to Central’s bots.
Meanwhile, Gumption and Dockety seek help from a community of free bots in the forest. A reclusive bot called the Tailor disguises Gumption as a machine, and they reluctantly join forces with Patient. Inside Heck, Maddy finds her father, but is captured and placed in a virtual reality. There, Central reveals it is trapped by its own programming and longs to end.
The climax takes place in Central’s laboratory. Gumption’s disguise lets him briefly command Central’s bots, but Patient seizes control of Central through Sindy’s network connection. Sindy defeats Patient, Thoughtful turns on Special Blazes to save Sindy, and Dockety is nearly destroyed shielding Sindy from harm. The group escapes Heck with Maddy’s father and a handful of other prisoners. As the story ends Papa Roo dreamily announces that the Makers are awake and returning.
The Makers Return
The Makers Return picks up several months after In Heck. Special Leaf has died and left his house and his collection of ancient tech to Sindy. With Maddy and Gumption away in Sandy Bend, she is struggling to find her place in Rusty Bridge. She discovers a communicator that connects her to Violet, a young human girl aboard a failing spaceship called the Ark that has lost contact with Central and is running out of fuel. The ship’s commander, Captain Leung, decides it is time to return to the planet.
Special Blazes returns to Rusty Bridge with a new partner just as the Ark crash-lands in the swamp, where it is seized by a tentacled bot first encountered in Maddy Roo. Sindy uses her abilities to command the bot to release the ship, but a second attack creates chaos. Captain Leung seizes Sindy and, with Violet, flees in a shuttle. When the shuttle is forced down near an abandoned bunker, Patient in Darkness is waiting for them. The mad bot captures the group and uses a neural cap to read Captain Leung’s memories, confirming that the Makers have truly returned.
Violet discovers that Patient plans to use the Makers’ combat bots to destroy the ark. She escapes with Sindy and Mouse, and lead Patient (in a giant new body) back to Rusty Bridge. There, Violet wields a remote-control glove from Special Leaf’s hidden cache to disable Patient’s forces, and Mouse convinces the swamp creature to drag Patient under the water for good. In the conclusion, we see Violet and other children from the Ark settling into Rusty Bridge as Captain Leung calls the other arks home.
I’ve been unemployed for eight months now,
and haven’t written as much as I thought I would.
Middle-agedangst is one reason,
but another is the realization that
most of the projects I was thinking of doing are solving yesterday’s problems.
I have a long history of doing this:
The JavaScript and Python versions of
Software Design by Example
are the books I needed in the 2000s when I was teaching undergraduate courses
at the University of Toronto.
I’m proud of them,
but they have essentially found no readers.
Similarly, Research Software Engineering with Python
would have been really useful if it had appeared in 2012 or 2013.
By the time it came out in 2021,
most of what it said was already online in a hundred places
(thanks in part to Software Carpentry).
By the time we shut down It Will Never Work in Theory
I had learned enough to write an introductory textbook on software engineering
that would show readers what we actually know about software development
and (just as importantly) why we believe it’s true.
I think that a course like that would have prepared people to tell
whether AI is making them more productive or not,
but as I wrote six years ago,
the people who teach undergrad SE courses don’t seem to be interested in changing the curriculum.
Which brings me to the projects I’ve been noodling with since November:
workshops on organizational change,
project closure,
and managing research software projects,
and tutorials on debugging, Gleam,
and a few things programmers ought to know about how society actually works.
According to page views on Plausible,
none of these have had more than a couple of dozen viewers a month,
and when I ask people for feedback,
I hear crickets.
As someone who believes we ought to teach young programmers to pay attention to evidence,
it’s hard for me to ignore these signals;
as someone who has written several books that (to first order) nobody read,
it feels foolish to do so.
So I’ve been trying to write fiction instead,
which has its own frustrations.
Publishers are drowning under AI slop,
so most won’t accept unagented submissions any longer,
but agents are drowning as well.
(There is also the fact that my fiction might not be as good as I think it is:
feel free tojudgeforyourself.)
I’ve tried self-publishing a couple of times in the past;
the results made sales of my technical books look stellar.
Which leaves me looking at two half-finished YA novels and a pile of non-fiction essays,
and wondering if any of it is worth any more time.
A friend suggested that I put it all aside and devote myself to
Toronto Nature Stewards or some other volunteer work for a few months,
if only to get off the screen and meet some new people.
There’s also an election coming up;
city councillors are always grateful for IT help,
and working on a winning campaign has been on my bucket list for over thirty years.
Right now,
though,
I’m going to take another look at the outline for one of those stories
and hope that inspiration strikes.
Time for another cup of tea.
If you came in peace, be welcome.
A lot of people are afraid that AI is going to take their jobs.
That fear is legitimate:
it’s what happened in agriculture when mechanization arrived in the nineteenth century,
to craft manufacturing when factory automation took over,
and to office work when computers eliminated most routine clerical jobs.
New work appeared,
but it wasn’t the same work or in the same places,
and it wasn’t for everyone who needed it.
I think understanding that history is essential to understanding what’s happening with AI.
What AI Is, and What Automation Does
Large language models are trained on enormous quantities of text and code,
almost all of which was produced by people who weren’t paid and didn’t consent.
LLMs generate statistically plausible outputs:
they don’t understand what they’re saying,
and can’t verify that their output is accurate
or distinguish confident nonsense from correct reasoning
because they don’t reason [Torres2024].
In December 2023,
the New York Times filed suit against OpenAI and Microsoft,
arguing that training a commercial AI system on millions of copyrighted articles without permission
constituted infringement.
It was the first major legal test of that question,
and courts in multiple countries are working through similar cases.
As described earlier,
intellectual property law has always been an arena where
the better-resourced party has a structural advantage.
Whatever precedents emerge from these cases will reflect
who could afford to pursue them to conclusion,
not some Platonic ideal of “right”.
None of this makes AI unusual as a technology.
Manufacturing employed 19.4 million workers in the US in 1979.
By 2023 that had dropped to 12.8 million.
The jobs that replaced the ones that automated or were shipped overseas
were often in different sectors or different regions,
and almost always lower paid.
Towns built around steel mills or textile factories didn’t reinvent themselves as technology hubs:
they lost population, services, and tax base simultaneously,
and many have not recovered.
The economist Daron Acemoglu estimates that
roughly half of the increase in US income inequality since 1980
can be attributed to automation
that systematically replaced mid-wage workers with machinery and software [Acemoglu2023].
The gains from automation go to whoever owns the tools,
while the cost of retraining,
years of lower wages,
and the disruption of moving somewhere else
fall on the workers who are displaced.
Every previous wave of automation distributed those costs unfairly as well,
not because it had to,
but because the people who owned the technology
had more political power than the people displaced by it.
The Demand Problem
The economic structure of AI displacement creates a specific problem
that economists Brett Hemenway Falk and Gerry Tsoukalas call “the AI layoff trap” [HemenwayFalk2026].
In competitive markets,
an automating firm captures the full cost savings from replacing workers
but bears only a fraction of the resulting demand destruction.
In a market with twenty competitors,
each firm absorbs one-twentieth of the demand it destroys;
the rest falls on rivals.
Every firm therefore has a rational-as-in-psychopathic incentive
to automate beyond the socially optimal level,
because the gain from cutting labor costs outweighs
the diffuse shared consequence of eliminating consumer spending.
AI worsens this: wider productivity gains accelerate the race toward a shrinking market.
Ironically,
Henry Ford (no friend to workers) understood the opposite logic:
his employees needed to earn enough to buy his cars.
The AI economy is eliminating the workers and expecting the cars to keep selling [McGrann2026].
Sometimes the layoffs happen before anyone checks whether the technology can do the job.
Acemoglu’s term for this is “excessive automation”:
using AI to eliminate jobs without generating meaningfully lower production costs,
while imposing substantial social costs.
When Block’s Jack Dorsey laid off nearly half his workforce in March 2025,
citing AI coding agents,
investors responded by boosting Block’s stock price by twenty-five percent.
The market rewarded the elimination of human labor with an immediate transfer of value to shareholders,
regardless of whether the AI actually performed the eliminated work.
In the Long Run
Anne Case and Angus Deaton tracked what happened to communities when manufacturing employment disappeared.
The answers were grim:
rising rates of suicide, drug overdose, and alcoholic liver disease
all increased among people who had lost their economic function [Case2021,Suzman2021].
The mechanism was not only poverty but the loss of purpose, social status, and a perceived future.
As noted above,
communities organized around industries that left did not quietly transform into something else.
The AI industry’s narratives about abundance repeat the promises of globalization.
The evidence from globalization is that the losers do not become winners on their own,
and their losses produce political consequences that outlast any particular trade agreement.
AI tools are also degrading the workers they are supposed to help.
Anthropic’s own internal research found that junior engineers
who relied heavily on AI coding agents understood their work significantly less when tested afterward,
even though they completed tasks at roughly the same speed as those who did not.
The retraining argument assumes people can develop new skills to stay relevant.
The evidence suggests that the tools accelerating displacement
are simultaneously eroding the capacity for skill development.
What makes me really angry is that
the research underlying this technology was publicly funded.
The mathematical advances, training methods, and semiconductors
were developed through universities, DARPA, and national laboratories,
but private companies captured the reward.
As Mazzucato has argued,
invention has become an engine of rent extraction rather than value creation [Mazzucato2013].
We’re now speed-running that process.
By the first three quarters of 2025,
AI-related investments accounted for roughly thirty-nine percent of US economic growth,
giving the federal government a vested interest in sustaining the boom.
The interventions that economists have identified,
such public ownership stakes in AI infrastructure,
aggressive antitrust enforcement,
and a tax on automated labor,
are what people in public health call “abstinence solutions”:
they would work if people actually implemented them,
but we know that’s not going to happen.
The Business Model and the IP Problem
AI services are currently cheap or free,
but that can’t last.
OpenAI lost approximately $5 billion in 2024 providing cheap API access.
The cheap phase exists because companies are burning investor capital to capture market share
and deprive competitors of users.
This is enshittification all over again:
attract users with artificially low prices, build dependencies,
then raise prices once alternatives have been squeezed out.
The useful, affordable version of these tools will not survive for long,
and the developers, writers, and companies that build workflows around them
during the subsidized period
will pay for it later.
The intellectual property question adds a separate layer of instability to the whole enterprise.
Writers, artists, musicians, and software developers whose work was ingested
to train commercial AI systems
received neither payment nor credit for that contribution.
Whether this constitutes infringement, fair use, or something else entirely
is actively contested in courts across multiple jurisdictions.
The outcomes will depend partly on how judges read copyright law
and partly on which side has the resources to sustain litigation that may take a decade to resolve.
The largest AI companies have substantially more resources than the individual creators suing them.
Ransomware attacks demonstrate how extortion,
if professional enough,
is indistinguishable from any other fee-for-service arrangement.
The 2017 WannaCry attack encrypted hundreds of thousands of computers across 150 countries in a single weekend.
Four years later,
the DarkSide group shut down the Colonial Pipeline and demanded approximately $4.4 million in Bitcoin;
the company paid within hours.
Modern ransomware groups operate on an affiliate model—core developers write the malware,
affiliates handle intrusions—and cybersecurity firms handle negotiations
the same way kidnap-and-ransom specialists did for physical abductions in the 1990s.
Both sides have an interest in the transaction completing cleanly.
Governments officially discourage paying ransom
while intelligence services routinely help to do exactly that.
Cyber insurance policies now cover ransom payments,
and insurance companies are wrestling with moral hazard and ransom inflation—
the same concerns Lloyd’s of London was managing thirty years ago
[Dudley2022].
The Standard Playbook
Major AI companies have not waited for regulators to define rules that might constrain them.
They have placed former employees and allies in regulatory positions
and submitted their own proposed frameworks to legislative consultations.
For example,
when the European Union was developing its AI Act,
Anthropic, Google, and OpenAI all submitted proposals
that would have exempted their most powerful models from the Act’s strictest requirements.
AI laboratories have also funded their own safety research and publicized favorable results.
Critics of AI development have been characterized as alarmists,
and documented harms have been described as edge cases.
When OpenAI’s safety team resigned in 2024,
several members stated that commercial considerations had systematically overridden safety commitments.
This sequence—fund your own science,
frame independent critics as emotional rather than analytical,
and describe any harm as an unfortunate anomaly—is the same one used by tobacco companies
and the producers of leaded gasoline.
The reframing of displacement as individual opportunity is equally familiar.
The slogan “AI won’t replace you; someone using AI will”
shifts the burden of adaptation entirely onto workers
and treats the costs of corporate automation as a personal problem requiring a personal solution.
This is the passion principle applied to survival:
workers are told to reskill and stay relevant,
rather than that the economy owes them any compensation for a transition they did not choose.
The same framing accompanied every previous major automation wave.
What Collective Action Has Achieved
In 2023,
the Writers Guild of America struck for five months over issues that included AI.
When the strike ended, the WGA had won explicit contract language:
AI cannot write or rewrite scripts,
and scripts cannot be used to train AI systems.
The Screen Actors Guild reached a parallel agreement
that included restrictions on the digital replication of performers’ likenesses
without ongoing consent [Kelly2022].
These victories established enforceable contractual limits
on what employers could do with AI—limits that individual workers negotiating alone could never have secured.
The lesson is not specific to Hollywood:
wherever workers have collective bargaining rights,
they can negotiate from a position of strength.
Professional associations, open-source communities, and standards bodies
can create analogous leverage in sectors where formal unions are absent or weak.
Regulation has also moved faster than the industry claims is possible.
The EU AI Act requires transparency for high-risk systems,
mandates human oversight for consequential automated decisions,
and bans specific applications outright.
Canada, Brazil, South Korea, and the United Kingdom all have AI governance frameworks in development.
Before the EU’s General Data Protection Regulation took effect in 2018,
industry associations described it as “unworkable”
and predicted that it would destroy European tech competitiveness.
By 2024 it had generated approximately $4 billion in fines
and had changed how companies worldwide handle personal data,
including companies with no European operations
that simply chose to comply rather than maintain two systems.
The argument that AI regulation will destroy innovation
has been made about every major technology regulation in living memory,
and has been wrong every time.
Alternatives to the dysfunctions described in this series of post exist.
Ranked-choice voting in Ireland, Australia, New Zealand, and more than a dozen US cities
has not produced chaos:
it has produced legislatures that more closely reflect what voters actually want.
Independent redistricting commissions have measurably reduced partisan gerrymandering
in Arizona, California, and Michigan,
where independent bodies now draw district lines rather than the legislators who benefit from them.
New York City’s public matching funds for small donations have shifted the incentive structure for candidates,
making it possible to run a competitive campaign on small contributions
rather than depending on a handful of major donors.
Broad constitutional reform through sustained public participation succeeded in Iceland
following the 2008 financial crisis.
Campaigns that engaged the active participation of roughly 3.5 percent of a population
have been sufficient to force political change in case after case.
Electoral organizing, legal challenges, constitutional campaigns,
and redistricting advocacy have all worked when that threshold of organized, sustained pressure was reached.
The rich and powerful will always resist;
while the specific tools differ,
sustained, organized pressure wins time after time [Young2024].
A Paradise Built in Hell
On the morning of December 6, 1917,
a French munitions ship collided with a Norwegian vessel in Halifax Harbour, Nova Scotia.
The resulting explosion killed nearly two thousand people and flattened the north end of the city.
It was the largest human-made explosion before the nuclear age.
Within hours,
survivors were pulling strangers from rubble,
improvising hospitals in churches and railway stations,
and sharing food with people they had never met.
The next day a blizzard arrived.
Residents of Truro, two hours away by train,
loaded relief supplies and medical teams before anyone had formally organized them.
People came from across eastern Canada and the northeastern United States,
not because anyone had issued orders,
but because other people needed help.
This is not the story most people expect.
The version of human nature embedded in popular culture
and reproduced in disaster media coverage
is that when things fall apart, so do people.
Civilization is a thin crust over barbarity:
scratch the surface and you get looting, assault, and the strong preying on the weak.
This story is wrong in almost every particular,
but it keeps being told because it serves purposes that have nothing to do with accuracy.
The sociologist E.L. Quarantelli spent decades studying disasters
and came to a conclusion that surprised many people:
panic and antisocial behavior are the exception, not the rule.
Communities typically show increases in prosocial behavior:
strangers help each other,
crime rates generally fall,
and people who were barely acquaintances briefly become something like a community.
Rebecca Solnit documented this pattern across a century of catastrophes.
Her case studies,
including the 1906 San Francisco earthquake,
the 1917 Halifax explosion,
the 1985 Mexico City earthquake,
the September 11 attacks in New York,
and Hurricane Katrina in New Orleans,
illustrate Quarantelli’s findings.
Disasters reveal a capacity for mutual aid
that is usually suppressed by the atomization of modern consumer society.
Hurricane Katrina in 2005 produced the most extensively documented divergence
between media narrative and documented reality in modern history.
In the days after the storm, major news organizations reported roving gangs in the Superdome,
mass rape,
and snipers firing at rescue helicopters.
Subsequent investigation found that the reported gang violence did not happen,
the murder rate in the city did not spike,
and most of the “looting” was people taking food and water to survive.
But these lies had consequences.
Hospitals delayed evacuating critically ill patients while waiting for military escorts.
Trucks carrying food and water were turned back from routes deemed dangerous when they weren’t.
A group of survivors trying to walk across the Crescent City Connection bridge to reach Gretna,
where they had been told buses were waiting,
were turned back at gunpoint by police who said they were keeping their community safe.
The fiction of social breakdown caused deaths that the storm itself had not.
Solnit has a name for what happened in New Orleans: elite panic.
Ordinary people in a disaster tend to behave with remarkable generosity and calm,
but authorities and elites tend to panic—not about the disaster, but about the public.
Since they believe that the social order that keeps them on top
is only held together by the threat of force,
a disaster that removes their ability to enforce their rules looks like the end of civilization.
The gap between what happens in disasters and what gets reported
is also explained by what counts as news.
Editors make decisions about what to show based on what will attract attention,
and dramatic conflict attracts more attention than organized mutual aid.
The result is systematic selection bias in disaster coverage.
If the media consistently describes human nature as more violent and more selfish than it actually is,
people are pre-conditioned to believe that cooperation is unlikely,
which makes them less likely to cooperate.
Just as advertising can manufacture demand,
biased reporting can manufacture mistrust,
and in doing so, hurt us all
[Quarantelli1998,Solnit2009,Tierney2006].
The Ozone Hole That Closed
In 1974,
two chemists at the University of California published a paper
predicting that chlorofluorocarbons (CFCs) would destroy the ozone layer in the stratosphere
that shields Earth from ultraviolet radiation.
CFCs were used as propellants in aerosol cans and refrigerants in air conditioners,
and while the paper’s authors didn’t yet have a hole to point to,
they had atmospheric chemistry on their side.
The chemical industry’s response was to fund counter-research,
hire lobbyists,
and describe the scientists as alarmists
whose work was too speculative to justify regulatory action.
This was the same playbook that the tobacco industry had been running for two decades,
and for a while it worked.
The Alliance for Responsible CFC Policy,
a trade group representing the manufacturers,
argued that the science was uncertain.
Industry representatives testified before Congress
that banning CFCs would cost hundreds of thousands of jobs
and devastate the American economy.
Du Pont,
which held a large share of the CFC market,
said in 1975 that it would stop making CFCs only if a worldwide scientific consensus emerged
and the appropriate regulatory bodies took action.
This was not a promise to act;
it was a description of conditions
the company presumably believed would never be met.
Eleven years later,
in 1985,
a team from the British Antarctic Survey
reported a massive and growing thinning of the ozone layer over Antarctica every southern spring.
Their data was so far outside expected ranges
that they initially assumed their instruments were broken.
NASA confirmed the finding using satellite data that,
embarrassingly,
had been sitting in archived files for years
after automated quality-control software had flagged the anomalous readings as errors
[Roan1989].
Industry resistance collapsed in just two years,
and the Montreal Protocol was signed in 1987.
The protocol’s design explains why it worked when so many other environmental agreements have not.
It set binding phase-out schedules for ozone-depleting substances,
with different timelines for developed and developing countries.
It established trade sanctions against non-signatories,
which meant that countries outside the agreement faced economic costs for staying out.
And it created the Multilateral Fund,
which transferred technology and money from wealthy countries to developing ones
to help them adopt CFC alternatives.
This last element is the one that gets least attention
and does the most work.
When the Montreal Protocol was negotiated,
China and India were skeptical.
Both were industrializing rapidly,
both had growing demand for refrigeration and air conditioning,
and both pointed out (reasonably enough)
that the damage to the ozone layer had been caused almost entirely by wealthy countries.
The demand that they now forgo the same technologies their economic competitors had used
looked like a way of keeping them poor.
The Multilateral Fund changed the calculation.
By 2023,
the fund had disbursed over $4 billion
to help developing countries transition away from ozone-depleting substances.
China became one of the largest recipients of technology transfer funding
and one of the most consistent compliers with phase-out schedules.
India followed a similar path.
Neither country did this because their leaders suddenly became environmentalists:
compliance became economically rational once the fund made alternatives affordable.
The protocol’s structure gave it leverage that most international agreements lack.
A country that refused to sign could not import controlled substances from signatory countries
and could not export products made with those substances to them.
By the early 1990s,
enough of the global economy was covered by the agreement
that staying outside it became genuinely costly.
Du Pont,
which had spent years arguing that alternatives to CFCs were technically impossible,
announced shortly after the protocol was signed
that it had developed workable substitutes
and would accelerate their commercialization.
What had been technically impossible became technically straightforward
once the regulatory framework made the old product unmarketable.
The substitutes developed to replace CFCs were hydrofluorocarbons—HFCs.
They did not destroy the ozone layer.
They did, however, turn out to be extremely potent greenhouse gases,
some of them thousands of times more warming per molecule than carbon dioxide.
In switching from one problem to another,
the world had traded an acute crisis for a contribution to a chronic one.
The Kigali Amendment to the Montreal Protocol,
adopted in Rwanda in 2016,
addressed this.
It added HFCs to the list of controlled substances
and set phase-down schedules for them as well.
Developed countries agreed to begin reductions by 2019;
most developing countries by 2024 or 2028,
with a small number of the hottest-climate countries,
including India and Pakistan,
given until 2032.
The amendment was negotiated under the same structure as the original protocol,
with the same Multilateral Fund available to support transitions.
Climate scientists estimated at the time
that full implementation of the Kigali Amendment would avoid
up to 0.4 degrees Celsius of warming by 2100.
This is not is a story about individual consumers making better choices.
Millions of people did not read scientific papers and switch to pump-action hairspray.
The mechanism was a binding international agreement
with differentiated obligations,
a technology transfer fund,
and trade sanctions against non-participants.
The lesson for climate change shouldn’t need to be spelled out,
yet it rarely appears in public discussions.
Renewable energy investment,
corporate sustainability pledges,
and carbon pricing mechanisms will all help,
but the decisive ingredient for the ozone layer was a binding agreement with teeth.
Similarly,
if we want to mitigate the cognitive pollution caused by social media,
country-by-country age verification isn’t going to make a difference
[Parson2003].
Land to the Tiller
In 1947,
the United States government did something
that its own politicians would have called socialism
if anyone else had done it.
Under American military occupation,
Japan’s agricultural land was seized from landlords
and sold to the tenant farmers
who had been working it,
at prices set well below market value,
paid in bonds that inflation promptly turned into confetti.
This was expropriation, and it worked.
The Cold War was the reason.
American planners in Tokyo feared that rural poverty and landlord domination
were exactly the conditions in which communist movements flourished.
They had watched what happened in China and did not want a repeat,
so they did what they would never have considered at home:
they redistributed productive assets
from the wealthy to the poor
on a massive scale
and called it democratization.
Between 1947 and 1950,
roughly thirty percent of Japan’s farmland
changed hands under the land reform program.
Landlords who had lived off tenant rents for generations suddenly held bonds
whose real value was eaten away month by month,
while the tenants who had always done the work owned the fields.
The landlord class as an economic force essentially ceased to exist.
What replaced it was a rural middle class of owner-farmers.
In the following decades,
those farmers’ children moved to the cities
and provided the workforce for Japan’s industrial expansion.
The land reform did not just change who owned the fields;
it restructured the society
that would industrialize in the 1950s and 1960s
[Dreze2013,Studwell2013].
South Korea and Taiwan followed the same template,
for the same reasons,
at almost exactly the same time.
In both places,
American advisors pushed land reform
as a counter to communist land redistribution programs
that were mobilizing peasant populations elsewhere in Asia.
In South Korea,
the Land Reform Act of 1950 capped landholdings
and required excess land to be sold to the state
for redistribution to tenant farmers.
In Taiwan,
the program between 1949 and 1953
transferred land from Taiwanese landlords to the tenant farmers who cultivated it.
The compensation paid to landlords in both countries
was structured in ways that made delay expensive:
bonds whose value eroded,
or equity in state enterprises
whose worth depended on economic policies the landlords no longer controlled.
The design was intentional.
Reform administrators understood
that the landlord class would use any instrument available
to reverse the transfer,
and they structured the compensation
to reduce the resources available for that reversal.
This was not incidental:
the land reforms created the conditions
for the subsequent industrial policies to succeed,
because the rural population had both the stability
and the incentive to participate in markets
rather than spending their energy surviving extraction.
Things went differently in Latin America.
Bolivia’s 1952 land reform and Guatemala’s 1952 program
under President Jacobo Árbenz
both attempted to redistribute agricultural land
in societies with high inequality.
Bolivia’s reform survived in partial form
but was repeatedly undermined by subsequent governments.
Guatemala’s program was ended in 1954
when the CIA backed a coup
that restored land expropriated from the United Fruit Company.
Chile’s reform effort under Salvador Allende
was reversed after the 1973 coup backed by the United States.
In each case,
the political conditions
that allowed redistribution to happen
were themselves unstable,
and the reform did not survive the removal of the government that carried it out.
The Japanese, Korean, and Taiwanese cases all share a feature that is easy to overlook:
the reforms were imposed from outside,
and so were insulated from the normal political power of the landlord class.
This raises an uncomfortable question
about whether the reforms could have happened through domestic democratic politics.
The state of Kerala, in southern India,
provides an answer.
Kerala’s land reform story begins with electoral politics
rather than military occupation.
The Communist Party of India won state elections in Kerala in 1957
on a platform that included land reform,
and despite being dismissed from power by the central government before completing its program,
it returned to power and passed the Kerala Land Reforms Act in 1969.
The legislation abolished tenancy arrangements
that had kept agricultural laborers in conditions of near-permanent dependency,
placed ceilings on landholdings,
and required excess land to be redistributed.
Landlords resisted,
courts were used to delay implementation,
and the process took years to work through,
but it worked.
The Kerala case is important because
it demonstrates that land reform can happen through democratic elections
in a country
where the landlords have full political rights
and access to courts and legal challenges.
Landlords resisted energetically,
but the political organization of tenant farmers and agricultural laborers
was strong enough and persistent enough
to sustain reform across multiple election cycles
and through sustained legal obstruction.
What makes Kerala remarkable is what happened afterward.
By the 1990s the state had achieved literacy rates, life expectancy, and infant mortality figures
that compared favorably not just to other Indian states
but to countries with far higher per-capita incomes.
Land reform broke the power of a class
that had used political dominance to block public investment in health and education;
once that class’s power was broken,
public services became possible.
The words “land reform” have also been used to describe something very different.
Stalin’s forced collectivization of Soviet agriculture between 1929 and 1933
drove peasants into collective farms at gunpoint,
killed or deported millions of people labeled “kulaks” for owning a cow or two,
and caused a famine
that killed somewhere between five and eight million people in Ukraine alone.
Agricultural output collapsed for years.
Mao’s collectivization in China followed the same blueprint with even worse results.
The Great Leap Forward of 1958 to 1962
forced peasants into communes,
requisitioned grain from villages even as harvests failed,
and caused a famine that killed an estimated thirty to forty-five million people.
These programs had nothing in common with the reforms described in this lesson.
Japan, Korea, Taiwan, and Kerala gave farmers ownership of the land they worked.
Stalin and Mao abolished private ownership entirely
and replaced it with state control enforced by violence,
combined with the systematic destruction of any incentive to grow food.
Critics who invoke collectivization to argue against democratic land reform
are comparing policies that created owner-farmers with policies that destroyed them
[Conquest1986,Walder2017].
The argument made against land reform in all of these cases
was that it would destroy productivity,
undermine investment incentives,
and leave everyone worse off.
Big tech makes the same arguments today
about proposals to democratize social media and break up virtual monopolies.
There is no reason to believe the outcomes would be different
[Studwell2013].
Conclusion
These essays have described how power is structured, how harm
is produced and obscured, who bears the costs, and how regulatory and
political contests have unfolded in other industries. This final
lesson asks what the historical record shows about how change actually
happens in documented cases rather than in theory. The answer is
consistent across domains and largely unwelcome to people who prefer
to change the world through individual choices or technical solutions:
change happens when organized groups apply sustained economic and
political pressure over time, and it rarely happens any other way. The
record also shows that nonviolent campaigns have historically been
more successful than violent ones, and that the reasons why are
structural and replicable.
The dominant popular narrative about social change centers on individuals:
Rosa Parks refused to give up her seat and the Civil Rights Movement was born.
This narrative is factually wrong and strategically disabling.
Rosa Parks was the secretary of the Montgomery chapter of the NAACP
and had recently attended the Highlander Folk School,
a training center for labor and civil rights organizers.
The Montgomery Bus Boycott that followed her arrest was organized by the Montgomery Improvement Association,
coordinated carpools across a city for over a year,
and was sustained by the labor of hundreds of people whose names are not remembered.
The choice of Parks as the plaintiff in the subsequent legal case was deliberate:
other potential plaintiffs had been rejected as less strategically suitable.
This is what organized political campaigns look like.
The reduction of that campaign to one person’s spontaneous act of courage
makes it both more inspiring and less useful as a model
[Beckerman2022].
The historical record of successful social change campaigns shows
consistent structural features that cut across very different political contexts.
Indian independence was achieved through a disciplined mass movement
that combined civil disobedience,
economic disruption,
legal challenge,
and international publicity over decades.
Polish Solidarity built an independent trade union into a national opposition movement
that eventually outlasted the communist state,
sustained through martial law and repression by organizational capacity and international support.
The South African anti-apartheid campaign combined internal mass action
with an international sanctions and divestment campaign
that imposed economic costs the apartheid government could not absorb indefinitely.
The British suffragette movement used tactics ranging from
petitioning and public speaking to window-smashing, arson, and hunger strikes,
and it succeeded only after the combination of sustained pressure
and the changed political calculus produced by women’s wartime labor
made continued denial of the franchise politically untenable.
These campaigns differ in tactics, duration, context, and outcome.
What they share is organizational discipline,
sustained commitment across setbacks,
and an understanding of where the economic and political pressure points lay
[Chenoweth2011,Lakey2018].
The most rigorous quantitative analysis of this question
is Erica Chenoweth and Maria Stephan’s study of 323 resistance campaigns between 1900 and 2006.
Their finding is that nonviolent campaigns succeeded roughly twice as often as violent ones,
and that the threshold for success was consistent:
campaigns that engaged the active participation of roughly 3.5 percent of the population did not fail.
The mechanism is not mysterious.
Nonviolent campaigns can recruit from a broader population,
including people who will not take up arms but will march, boycott, strike, or withdraw labor.
Broader participation creates broader legitimacy
and makes it harder for the state to frame repression as protecting order
rather than suppressing dissent.
The 3.5 percent figure is not a guarantee;
it describes a historical pattern.
But it is a more useful starting point than the assumption that
popular majorities produce change automatically.
Economic disruption is the mechanism that connects organized pressure to actual policy change.
Boycotts raise the cost of doing business with a target.
Strikes remove the labor on which production depends,
and divestment campaigns raise the cost of capital
for targeted firms or governments and create reputational pressure on institutional investors.
The Montgomery Bus Boycott worked because it destroyed the bus company’s revenue from its Black ridership.
The South African divestment campaign worked because
it raised the cost of the apartheid state’s international borrowing
and created political problems for governments whose pension funds held South African assets.
In each case the mechanism was economic:
the people in power faced a cost-benefit calculation that changed.
Moral suasion may have affected some individuals.
It did not change the structural calculation that drove policy.
Moral arguments have a poor track record as the primary lever of social change,
and this fact is frequently misunderstood.
It is not that moral arguments are irrelevant:
they build coalitions,
provide the normative framework that justifies what a movement is asking for,
and affect the willingness of potential participants to accept personal costs.
But moral arguments addressed to those in power,
without the economic or political pressure that makes their rejection costly,
consistently fail.
Slaveholders did not free enslaved people because they were persuaded that slavery was wrong.
The tobacco industry did not voluntarily stop marketing cigarettes to children
because public health advocates published articles about harm.
Corporate privacy practices do not change because researchers demonstrate the extent of surveillance.
What changes the behavior of those who benefit from a harmful arrangement
is when not changing becomes more costly than changing,
and that calculation is economic and political.
The word “political” is consistently used disparaginly in tech culture,
as if politics were something that happens elsewhere
and that a well-run technical organization can avoid.
Politics is the process of making collective decisions in the absence of agreement on goals.
Every decision about which features to build,
which users to prioritize,
and which harms to accept does this.
The refusal to engage with questions framed as political
does not remove politics from the process;
it delegates those decisions to whoever is willing to engage.
Refusing to vote is a political act.
Refusing to join a union is a political act.
Choosing to work on a product without asking who it will harm is a political act.
The only question is whether the political choices being made are made consciously
and with an understanding of their consequences
[Young2024,Chenoweth2011].