How to Write a Technical Book (Part 2)

Following up on the first post in this series, here are a few more tips for anyone thinking about writing a technical book.

1. Read, then write. Go through your two favorite technical books and make notes about what their authors have done well. Doing this will give you some ideas for your own book, but it will also help you become more conscious of your writing—in musical terms, help you learn how to listen to yourself while you’re playing. I learned a lot by reading Brian Kernighan and Jon Udell, but your tastes may vary.

2. Do the diagrams on a whiteboard first. It’s faster and more flexible than any computer drawing tool (though I admit I don’t have much experience with fingertip sketching apps for tablets). The only downside is that you can’t save a whiteboard drawing and reload it later, but they’re so fast to re-create that that’s not an issue.

3. Don’t write a general introduction unless you’re going to be one of the first two or three to market. Instead, focus on a particular aspect (“Python for Web Scraping”) or a particular audience (“R for Sports”). Your potential audience may be smaller, but you’ll reach a much higher percentage of that audience, and a focused book is a lot easier to write.

4. Don’t try to be funny. Very few jokes are funny the second time you hear them; even fewer can stand a third re-telling unless they’re very, very dry. And on a related note, please don’t use exclamation marks: what you’re writing probably isn’t surprising, and certainly won’t be the second time around¹.

5. Avoid banal advice. Few things are as frustrating to the reader as sentences like, “You should carefully consider users’ needs,” because no sensible person would recommend the opposite. If you can’t provide a checklist of specific things to consider, or a scenario to give the reader insight into how you think through that class of problem, you’re giving them the intellectual equivalent of junk calories.

6. Don’t try to compete with the internet. Reference manuals were invaluable to me forty years ago, but serve little purpose when the world’s knowledge is just a click away. You add value by explaining the how rather than recapitulating the what, so do that.

7. Check your facts. It doesn’t matter how well you know your chosen subject—something will have changed since the last time you looked. And even if it hasn’t, it will in the year or more it takes you to write your book.

8. Automate, but proofread. This is the technical author’s equivalent of “trust, but verify”. I can re-run all of the examples in my latest book with a single command and capture their output and insert that output in the manuscript, but I still have to re-read the discussion around those examples to make sure it hasn’t fallen out of sync.

9. Enlist at least two people to do detailed reviews. Lots of people will do cursory reviews or give your manuscript a single careful read; people who will look at half a dozen sets of changes and see what’s actually on the page in each are worth their weight in rubies.

10. Stop and ship. Your book will never be perfect. It will probably never even feel finished, any more than software does. Ship it anyway.

1. And remember: nobody reads footnotes.

Dev Ed Book Club

Flattered and grateful that the Dev Ed Book Club has chosen Teaching Tech Together as their next book: see this page for details. Discussions will start online on July 7, 2022.

How to Write a Technical Book

I have written three technical books (with one more in the works), co-written three more, and edited six. Here’s what I’ve learned along the way:

1. Don’t write a book, at least not at first. It’s a big undertaking, and it’s easy to be discouraged. Instead, start blogging regularly: one article a week, each about a thousand words long. You don’t have to write them in the order you think they’ll eventually go into the book—in fact, one of the reasons to blog is to get a better idea of what you want to say—but you should try to stitch them together in multi-part sequences that can each eventually become a chapter.

2. Don’t expect to make a profit.* Your publisher will give you anything from 12% to 50% of revenue in royalties; assuming a US$50 price tag and sales of a thousand copies, the most you can reasonably expect is US$25,000. It’s going to take at least a few hundred hours to write the book, so in almost every case you’d be better off doing some contract coding.

   So why both writing?

   1. To build your reputation. Several famous technical publishers lose money on the books, but make it back because they can charge more for the classes and workshops you teach because you’re famous.
   2. To give back to the community. None of us got here without help, and I think we have a moral obligation to repay that by helping others.
   3. Because you enjoy writing. Some people like working with wood or wool; you might like working with words.

3. Start with one or two closely-aligned learner personas. A learner persona is a short description of who you’re trying to teach, what they already know, and what they want to learn. Create one or two that are fairly similar and write for them, because a book that’s meant for everyone is actually useful to no-one. And keep in mind the difference between a novice (who is trying to build a mental model), a competent practitioner (who has one and wants to fill in gaps in their knowledge), and an expert (who wants higher-level discussion of tradeoffs and alternatives): no single book can serve all three well.

4. Don’t write just to cover your ass. I’ve read or reviewed dozens of books that start with a short introduction to XYZ; I cannot remember a single one that was actually useful. (If your audience knows Python, they don’t need a chapter-length intro to the language. If they don’t, a one-chapter intro isn’t going to help them.) I eventually realized that authors included those short introductions so that they could claim the book was suitable for novices when they knew in their hearts it wasn’t. Please don’t do this.

5. Try it out in pieces. I was once part of a team that spent a year and a half building a product before showing it to any potential users. To no-one’s surprise except ours, it turned out that most of what we had written was irrelevant to their actual needs. Similarly, if I had run a workshop or course based on the material for my first book while I was writing it, I would have realized that one of my core assumptions was completely wrong while there was still time to correct it. Conferences are always looking for tutorials, your colleagues will probably welcome some lunch & learn sessions, and if all else fails you can create a YouTube channel to find out how your material sounds.

6. Plan to iterate. There’s no point trying material out if you don’t incorporate what you’ve learned, and no matter how carefully you plan, you’ll realize something about the content of Chapter 3 while writing Chapter 4 or 5. At a guess, every line of SDXJS has been rewritten at least twice, and a quick glance in the version control log tells me that some parts have been revisited over a dozen times.

7. Drywall, then paint. In other words, get the words and code down, then worry about diagrams, bibliography citations, glossary entries, and so on. The word FIXME appeared over a hundred times in first draft of T3; there’s no easy way to count, but I bet that I changed or discarded more than half of those by the time the text settled down.

8. Accept that all available authoring tools suck. I’ve used LaTeX, Microsoft Word, Markdown, and two flavors of computational notebook, and they’re all more frustrating than they could be. I’ve written and spoken about this many times, so I won’t belabor the point here, but please don’t blame yourself.

9. Remember Wilkie’s Law, which states that an author’s job is to produce the manure in which an editor grows something worth reading. You sweated over that paragraph, but your audience doesn’t see your effort, they see your results. And yes, the explanation seems clear, concise, and elegant to you, but you are not your readers. By definition, your editor and reviewers are right when they say, “This doesn’t make sense.” Please accept that and revise accordingly.

10. Don’t be afraid to set it aside. I wrote an introduction to R for Python programmers that will probably never see the light of day, and have stopped work yet again on an undergraduate guide to being a compassionate programmer. In both cases I lost faith in the project: the more I wrote, the less I believed the book would actually help its intended audience. I hope I’ll get back to the second book eventually, but I won’t feel ashamed if I don’t: scientists don’t expect every hypothesis to turn out to be true, and authors shouldn’t either.

One last word of advice: You’ll spot a typo within seconds of receiving the first printed copy, someone you’ve never met will say something uncomplimentary in a review online, and Chapter 9 will never be as good as you wanted it to be. To hell with all of that: be proud of what you built and of who you helped by building it.

Mitigation

Here’s a conventional view of roles within a small-to-medium tech company:

Here’s another way to look at them:

The second formulation seems to be more compelling, especially to researchers and programmers who haven’t worked in structured organizations before. They tend to regard anything that isn’t science or coding as “overhead”, but that’s as inaccurate as calling the regulatory parts of your genome “junk DNA”. Explaining those coordinating roles in terms of the disasters they prevent seems to make them more palatable.

The Wes Mongtomery of Software

I had a conversation yesterday with a data scientist who is now working for a large IT consulting company. We got to talking about the tension between the scripts that data scientists cobble together to get a particular answer and the multi-level cloud architectures with complete integration test suites that software engineers believe are the “right” way to build software that routinely makes hundred million dollar decisions. In my mind the argument about “hacked-up scripts” versus “over-engineered collections of microservices” arises from a conflation of two distinct ideas; since quadrant diagrams are almost always a sign that someone’s trying to sell you bullshit, here’s mine:

The Beatles’ version of “Twist and Shout” is harmonically, melodically, and rhythmically much simpler than Coltrane’s 1967 recording of “My Favorite Things”, but both tracks were recorded in a single take. In contrast, Ravel spent months rewriting his “Pavane for a Deceased Princess” and Beethoven spent almost two years tearing his Ninth Symphony apart and putting it back together. While every single note of each was carefully considered, the former is once again much simpler than the latter.

But here’s the thing: Coltrane and the Beatles were able to create something great in a single take because they didn’t actually do it in a single take: they had both played those particular songs many (many) times before doing it in the studio. Similarly, if you watch Bruce Springsteen and the E-Street Band play You Never Can Tell for the first time, they pull it off because they’ve played songs like it literally thousands of times.

I think the same can be true of software. I can’t count the number of tools I’ve written over the last forty years to transform text from one format to another; asking me to write another is like asking a seasoned session musician to play a twelve-bar blues, or asking the data scientist I was speaking to yesterday to fit a linear model to some time-series data. We don’t need to write design docs or unit tests this time because we’ve worked through the problem and its variations so many times before that our fingers are just going to find the right chords.

Expertise doesn’t emerge without reflective practice: if you want to be Wes Montgomery, at some point you have to lock yourself in your room and play standards for a year. I wish more people who program and do data science knew that; more, I wish the people who hire them knew it too and gave them the time to practice so that they could do something great in just one take.

Shake it up baby…

A Language for Teaching

I’m hoping to send Software Design by Example to the publisher by the end of this month, and it has me thinking once again about what a programming language designed for teaching ought to look like. Here’s one request:

Built-in support for incremental exposition of code.

Most good books on programming interleave exposition and code in ways that most languages don’t directly support. For example, authors commonly want to write something like this to give readers a roadmap for what’s coming next:

class Grid:
    ...constants...

    def __init__(self, size):
        ...set up...

    def fill(self, value):
        ...fill entire grid with value...

    def adjacent(self, x, y):
        ...return neighbors of (x, y)...

They then want to fill in those markers one at a time further down the page or a few pages later:

    def fill(self, value):
        for i in range(self.size):
	    for j in range(self.size):
	        self.cells[i, j] = EMPTY

I don’t know any programming language that allows me to write this as shown. Some “simple” text processing will allow me to write something like this in my source file:

class Grid:

    ## [+fill]
    def fill(self, value):
        ## [-fill "fill entire grid with value"]
        for i in range(self.size):
	    for j in range(self.size):
	        self.cells[i, j] = EMPTY
        ## [-fill]
    ## [+fill]

and then slice the marked regions to produce the two versions shown above, but having used (and built) several such systems, I keep wondering why we don’t just add this to the language itself. Literate programming promised this, and while I was a zealous user for a couple of years in the late 1980s, bolting LP onto pre-existing languages proved too clunky to catch on. And yes, there are tricks like the Blank Maneuver and tools like jdc for the Jupyter notebook, but the former confuses novices (“Wait, you’re deriving a class from itself?”) and the latter doesn’t support forward markers in the original definition to show where the later code is going to go.

This issue may seem pretty esoteric—after all, most programmers don’t write books—but it highlights two larger points. The first is that most programmers do have to explain the work at some point, and there’s precious little in-language support for doing that. The second point is that languages don’t have any other support for incremental exposition either. For example, every textbook has diagrams, but you can’t put those in your source code: Jupyter notebooks and R Markdown files can show you the plots produced by your code in situ, but they won’t let you draw things by hand.

So here’s my suggestion for an enterprising graduate student who wants to change the world: pick half a dozen books on programming and go through them to create a catalog of explanatory techniques. Once you have that, extend your catalog by looking at slide decks and videos of whiteboard talks, and then design a little language and editor with built-in support for the top N techniques: really built-in, not wedged into specially-formatted comments or requiring extra compilation steps to see what readers are going to see. The language itself could be as small as Quorum, Hedy, or Lox: it’s just there to give users something to explain.

I often ask people what they would work on if they could work on anything. Variations of this idea have been on my list for two decades; I don’t think I have enough years left to see it through myself, but I’d be happy to chat with anyone who wants to take it on.

Four Books I'm Not Writing (Plus One)

I have bits and pieces of several overlapping technical books right now, but can’t decide which if any to complete:

1. Building Software Together is advice for undergraduates working on their first team project (either in school or in industry). It’s the furthest along, but it’s been a few years since I was working with undergrads in large numbers so I suspect some of my advice is out of date and out of touch.

2. Data Science for Software Engineers is an introduction to data analysis that (a) spends as much time on the messy business of getting, cleaning, and presenting data as it does on statistics and (b) uses software engineering data for all its examples. This one is more interesting to me personally, but it would be a lot of work to complete, and based on the polite disinterest I’ve received every time I’ve tried to get support for it, I suspect the audience is smaller than I’d like it to be.

3. Designing Research Software is simultaneously a continuation of Research Software Engineering with Python, a book-length expansion of “Twelve quick tips for software design”, and a re-imagining of Software Design by Example (formerly Software Tools in JavaScript). Each chapter designs, constructs, and critiques a small version of something a research software engineer might actually build: a discrete particle simulator that illustrates the principles of object-oriented design, a file manager for large datasets that does error handling properly, a pipeline framework that records provenance in a reproducible way, and so on. Again, it’s interesting to me personally, but I suspect the audience is small.

4. Software Engineering: A Compassionate, Evidence-Based Approach is the undergraduate software engineering textbook I think our profession needs today. Its starting points are (a) students should be introduced to the scientific study of programs, programmers, and programming and (b) now that we know how much harm software can do, we should teach prevention and remediation up front, just like the best civil and chemical engineering departments do.

The “which” part of my quandary is just my usual dithering; the “if any” goes a little deeper. Sales of technical books have been dropping steadily for twenty years: even ones as good as The Programmer’s Brain or Crafting Interpreters struggle to get through the noise. Textbooks have fared even worse, and not just because of publishers’ jam-today starve-tomorrow pricing models. I don’t think any book I could write today will ever reach as many people as Beautiful Code did 15 years ago. Maybe that shouldn’t matter to me, but it does.

And going even deeper, the two books I’m proudest of are two you’ve never read: Three Sensible Adventures and Bottle of Light. I wrote the stories in the first for my niece; it’s now out of print, but I have framed copies of the artwork up on the wall in my office and they’ve gotten me through some pretty bleak moments. The second, a middle-grade story about a world without light, is only available through one of Scholastic’s in-school programs for reluctant readers: I’ve tried periodically to buy back the rights, but so far the answer has always been “no”.

I really enjoyed writing both of them. If I could do anything with the years I have left I’d write more stories like the ones I loved growing up. I’d like to introduce you all to a cloudherd named Noxy (short for “Noxious Aftertaste”: the children in her village are given unpleasant names in order to discourage dragons from nibbling on them). I’d like you to meet a bookster’s apprentice named Erileine, a giant robotic dinosaur who may or may not be the original Santa Claus, and an orphaned clone with an odd knack for mechanical things trying to make ends meet in a post-Crunch Antarctica.

But all I’ve done in the last ten years is accumulate rejections, which makes it hard to keep writing. (And yes, I know about authors who got some-large-number of rejections before making a breakthrough sale, but I also know people who’ve won the lottery…) I know I should pick one of the above and get it over the line, but on a colder-than-spring-should-be Saturday morning, it’s all too easy to spend half an hour writing a blog post.

Family’s awake; time to make tea. Be well.

Later: the book I’d actually like to write is Sex and Drugs and Guns and Code, but I still don’t know enough and I’m not a good enough writer. Building Software Together and the software engineering textbook in the #4 spot above are partly attempts to smuggle some of those ideas past the defenses that people like my younger self have erected to protect themselves from feeling uncomfortable about themselves; one of the reasons I set BST aside was the realization that I’m not stealthy enough to pull it off.

Software Design by Example

I have moved my book-in-progress on software design from https://stjs.tech/ to a new home on this site at https://third-bit.com/sdxjs/: the old site’s name no longer matched the book’s title (which has changed from Software Tools in JavaScript to Software Design by Example), and I am trying to cut back on the number of small domains I manage. The source remains available at https://github.com/software-tools-books/stjs/; if all goes to plan, Taylor & Francis will publish it by the end of this year.

Tehanu

When I die, I shall breathe back the breath that made me live. I shall give back to the world all that I didn’t do. All that I might have been and wasn’t. All the choices I didn’t make. All the things I lost and spent and wasted. I shall give them back to the world. To the lives that haven’t been lived yet. That will be my gift back to the world that gave me the life I did live, the life I loved, the breath I breathed.

— Ursula Le Guin, Tehanu

12 Quick Tips for Software Design

My paper “12 Quick Tips for Software Design” has just been published by PLOS. I’m grateful to everyone who provided feedback, and I hope you find it useful.