Now What? A Workshop on Error Handling

⇐ previous
Posted

I finally have time to flesh out ideas for lessons that I’ve wanted for years. However, if I can’t find a way to send them back to 2006, there’s no point writing them: very few people read long-form tutorials about software these days. I’d still be interested in comments, though—figuring out what I would teach always helps me learn.

Overview

Topic: Error handling.

Audience: Senior undergraduates who are comfortable writing programs in Python and JavaScript that are hundred of lines long, know how to raise and catch exceptions, and are familiar with SQL, C, and the Unix shell, but have no experience building production-robust programs.

Format: seven 45-minute lessons with exercises.

1) What Can Go Wrong

Taxonomy drill
Given six short programs in Python, JavaScript, and C, each containing a different type of error, classify each error using the taxonomy above and explain whether the program’s response (crash, wrong output, hang, silent skip) is appropriate for a production context.
errno pitfall hunt
A short C function uses fopen, fread, and fclose and checks errno after each call. The function contains three bugs related to C-style error handling (e.g., ignoring a return value, checking errno too late, not distinguishing error from end-of-file). Identify each bug and propose a fix.
Failure brainstorm
Given a brief description of a web form that accepts a user’s name, email, and a file upload, then stores the data in a database, list every error that could occur, including errors the user causes, errors the network causes, errors the OS causes, and errors the program itself could cause. Compare lists with a partner and identify any category you missed.

2) Error Propagation and Recovery

Propagation rewrite
A Python function reads a configuration file, connects to a database using values from that file, and runs a query. The function is given with no error handling. Rewrite it so that errors at each stage are wrapped with context and propagated appropriately, then rewrite the equivalent in C using return codes, noting what is harder and easier in each style.
Swallowed exceptions
A code snippet contains three except: pass or equivalent constructs. For each, explain what failure is being hidden, what could go wrong as a result, and what the correct handling would be. At least one case should be a place where logging-and-continuing is wrong even though it feels safe.
Retry with backoff
Implement a retry decorator (Python) or higher-order function (JavaScript) that wraps a function call, retries up to N times on specified exception types, uses exponential backoff with jitter, and raises the last exception if all retries fail. Test it against a stub that fails a configurable number of times before succeeding. Discuss which HTTP status codes should trigger a retry and which should not.

3) Error Communication

Message rewrite
Five error messages from real or realistic applications are provided (e.g., a raw Python traceback shown in a web UI, a message reading “MySQL error 1045 for user ‘admin’@’localhost’”, a message reading “Your session token has expired and cannot be refreshed”). For each, write a user-appropriate replacement that is informative without leaking internals, and write a separate message suitable for the internal log.
API error schema design
Design the error response body for a REST endpoint that registers a new user. The schema must handle missing required fields, fields that fail validation (email format, password length), a duplicate username, and an unexpected server failure. Show example JSON for each case.
Login timing attack
A login function is provided that queries the database first and returns “User not found” immediately if the user does not exist, or hashes the password and compares it (taking ~200ms) if the user does exist. Explain the timing side-channel, fix it so both branches take the same time, and discuss whether this fix is always necessary or whether it depends on the threat model.

4) Logging and Observability

Logging retrofit
A function that processes uploaded CSV files is given with a single except Exception as e: print(e) handler. Add structured logging using Python’s logging module (or a JavaScript equivalent). Include appropriate log levels for different error conditions, add relevant context fields, distinguish expected errors (bad CSV format) from unexpected errors (disk full), and identify what was unobservable before your changes.
Security audit of log output
Three log excerpts are provided, each containing at least one security violation (e.g., a plaintext password, a full stack trace that reveals an internal file path and SQL query, or a user ID that allows enumeration of account existence). Identify each violation, explain what risk it creates, and show the corrected log entry.
Logging strategy design
A data pipeline reads records from an API, transforms them, and writes results to a database. The pipeline processes records in batches. Design a logging strategy: what events are logged at each stage, what fields each entry includes, what constitutes a loggable error vs. a metric, and how an operator would diagnose a job that completed without crashing but produced fewer output records than expected.

5) External Systems

Error handling retrofit for a pipeline
A script that downloads a JSON file from an HTTP endpoint, parses it, and inserts records into a SQLite database is given. It has no error handling. Add appropriate handling for HTTP errors (including distinguishing retryable from non-retryable), JSON parse failures, missing required fields, and database constraint violations. For each error type, decide whether to abort, skip the record, or retry, and justify the choice.
Transaction retry
A function executes a multi-statement database transaction and fails with a deadlock error. Implement retry logic that retries the entire transaction on deadlock, does not retry on constraint violations, limits total retries, and logs each retry attempt. Discuss why retrying only the failed statement, rather than the whole transaction, is wrong.
Atomic file write
A function saves user settings to a JSON file by opening the file, serializing the settings, and writing directly. Demonstrate two failure scenarios where this approach corrupts the file: interrupted write, and crash between open and close. Implement the temp-file-then-rename pattern and explain why it is safe even if the process is killed mid-write.

6) Concurrency and Async Errors

Silent thread failures
A Python script spawns ten threads, each of which downloads a file and writes it to disk. When a download fails, the exception is printed to stderr but the main thread sees all tasks as complete. Rewrite using ThreadPoolExecutor, collect all futures, and report which downloads succeeded and which failed, then introduce a deliberate failure in two of the threads and verify the errors are caught and reported correctly.
Promise.all to Promise.allSettled
A JavaScript function fetches data from five independent APIs using Promise.all. The whole function fails if any one API is unavailable. Rewrite it using Promise.allSettled so that results from available APIs are returned and failures are reported per-API. When is Promise.all’s fail-fast behavior preferable?
Deadlock identification and fix
A function managing a shared cache acquires cache_lock and then stats_lock in that order. Another function acquires the same locks in the opposite order. Trace through a scenario where both functions run concurrently and deadlock. Fix the bug using lock ordering. As a second fix, add a timeout to the lock acquisition and show how to handle the case where the timeout expires.

7) Resilience Patterns and Production Practices

Exercises

Circuit breaker implementation
Implement a CircuitBreaker class in Python or JavaScript that wraps a function. It should track consecutive failures, open after a configurable threshold, fail fast while open, and attempt recovery after a timeout. Write tests that simulate a dependency that fails, recovers, and fails again, verifying the breaker transitions through all three states correctly.
Single points of failure analysis
A diagram shows a web application with a load balancer, two application servers, one database, and one external payment API. Each component can fail independently. Identify the single points of failure, propose bulkhead and timeout strategies to limit the blast radius of each failure, and discuss the cost (infrastructure, complexity) of each mitigation. At what point does additional resilience add more complexity than it is worth?
Error path test coverage
A small data processing function is provided, along with a test suite with 100% line coverage on the happy path. Enumerate all error paths in the function (e.g., invalid input, missing file, network failure, and malformed response). Write tests for each error path using fault injection (i.e., mock the failing component). Measure what fraction of error paths were untested before, and explain why line coverage is a misleading metric for error-handling code.

Appendix: Exceptions

Categories: education, proposal, software-engineering