Blog

By Topic

academia anecdote aosa book-reviews career
community conferences data-science diversity education
ethics favorite fiction humanities-writing humor
management noticed open-source opinion personal
politics programming proposal research retro
software-carpentry software-engineering student-projects technical-writing workshops

By Year

2020 2021 2022 2023 2024 2025 2026
2010 2011 2012 2013 2014 2015 2016 2017 2018 2019
2004 2005 2006 2007 2008 2009

Recent

This Book Would Be Really Useful

I’ve often blogged about books I want to read that I don’t know enough to write; here’s another entry in that list:

I want ten chapters, each describing a study that an overworked VP Eng could feasibly implement to assess their software development team and practices.

In more detail:

So, who would write such a book? My first choice is probably too busy to do more than a single chapter (if that), but people who took part in It Will Never Work in Theory might find it interesting enough to contribute. I think it would be useful and timely, and a way to get researchers and practitioners to talk to each other. If this sounds like something you’d be interested in, please give me a shout.

Rationalizing the Command Line

The Unix command line is the most successful user interface ever built, and also the worst one we are stuck with. It was assembled over half a century, one tool at a time, by people optimizing for their own fingers on a keyboard. The result is overlapping responsibilities, inconsistent flags, and names that mean nothing to anyone who was not there when they were invented. That didn’t matter when the only consumers were people willing to memorize its quirks; it matters a great deal now that the principal consumers are people trying to understand what an LLM is doing on their behalf.

This post is a thought exercise: what a set of replacement tools that have more readable names and more consistent (and therefore more learnable) flags look like? This wouldn’t make any difference to the LLMs; the aim would be to make their actions more legible to users who aren’t command-line nerds.

One core design decision is to stop treating everything as a stream of bytes, which forces every tool to re-derive structure. Two modern shells have already done this. PowerShell built a pipeline that carries .NET objects, not text. A command like Get-Process produces a stream of process objects; the next command reads their typed properties instead of parsing columns. PowerShell draws a hard line between data and display: ConvertTo-Json and ConvertTo-Csv turn objects into data, while Format-Table and Format-List format objects for the screen and are display-only: they must be the last command in a pipeline, and piping their output into a data command produces garbage. PowerShell also standardizes a set of common parameters (-Verbose, -ErrorAction, -WhatIf, -Confirm) shared by every command. Its flaws are the verbose Verb-Noun names, the case-insensitive abbreviable parameters, and awkward gotchas like ConvertTo-Json defaulting to -Depth 2 and silently flattening anything deeper.

Nushell goes further by making typed tables the standard. ls returns a table of records; where, select, and sort-by reshape it by field name; to json and to csv serialize it; from json and from csv parse it back; into int and into string convert types, and so on. A pipeline like ls | where size > 10mb | sort-by modified | to json therefore never has to parse a text column. Nushell’s weakness is that it keeps the legacy command names and their inherited flags, so the surface is still cluttered even though the data is clean.

The design below is the synthesis: Nushell’s typed tables, PowerShell’s separation of data from display, neither system’s legacy names, and a standard way to interrogate tools to get their output schema.

Principles

Typed data, not text.
Commands pass tables (lists of records with named fields) and scalars, never ad-hoc text. Text is a rendering.
One verb per tool.
Each tool does one thing; each thing is done by one tool.
One flag, one meaning.
A flag means the same thing everywhere. Shared flags include --recursive, --ignore-case, --reverse, --human, --quiet, --force, --interactive, --help, --version.
Data and display are separate.
Serialization is an explicit, final step (to json), never a mode hidden inside a data command.
Schemas are declared.
Every producer documents the fields it emits, so other tools (not just LLMs) can discover what a table contains.
Three types of commands.
Every command is one of three things: a producer (emits typed data), a transformer (reshapes typed data), or a serializer (turns typed data into text). Pipelines run producers into transformers into a serializer.

Producers

tool emits replaces
list path {name, type, size, modified, mode, owner, group} ls, stat
find path {path, type, size, modified} find, locate
resolve path {path, canonical, target} realpath, readlink
read path the file as a list of line records, or a string cat, head, tail, tac, less, zcat
search pattern {file, line, column, text} grep, egrep, fgrep, zgrep
processes {pid, ppid, user, cpu, mem, command} ps
usage path {path, size, files} du
space {filesystem, size, used, available, mount} df
system {os, kernel, hostname, arch, uptime} uname, hostname
user {name, uid, gid, groups} whoami, id, groups
time {now, utc, epoch} date
environment {name, value} env, printenv, set
interfaces {name, address, mac, state} ifconfig, ip addr
connections {protocol, local, remote, state} netstat, ss
mounts {device, mountpoint, fstype, options} mount, df
logged-in {user, tty, host, since} who, w, last
checksum path {path, algorithm, digest} md5sum, sha256sum, cksum

Transformers

These are the heart of the design. They work on any table and are the replacement for a large fraction of the text tooling.

tool what it does replaces
select field… keep named columns cut, PowerShell Select-Object
filter predicate keep rows matching a predicate awk on columns, Nushell where
sort field… sort by named fields sort -k, sort -t
unique field… drop duplicate rows uniq, sort -u
group field group rows by a field awk grouping
join right --on field relational join of two tables join, comm
count count rows (or a field’s values) wc -l, uniq -c
sum, mean, min, max field aggregate a numeric field awk '{s+=$1}END{print s}'
map expression compute a new field awk '{print ...}'
head N, tail N take first or last N rows head, tail
replace pattern replacement substitute text in a field sed 's///', tr
flatten unnest a nested field jq flatten

filter and map need a small expression language. The design keeps it deliberately tiny—field access, comparison, arithmetic, and boolean combinators—because a full programming language bolted onto a filter is how awk happened.

Serializers

tool output replaces
to table aligned text table for humans column, fmt
to json JSON jq, ConvertTo-Json
to csv CSV ConvertTo-Csv
to plain one field per line, for text pipelines implicit text

Others

File and directory operations

new old
go cd
copy cp
move mv
remove rm,rmdir
create touch, mkdir with --directory, mktemp with --temporary
link ln
permit chmod, chown, chgrp via --mode, --owner, --group

Processes

new old
signal kill, killall, pkill
monitor top
run nice, timeout, nohup

System

new old
help man, apropos, whatis
which which, whereis, type

Network

new old
fetch curl, wget
connect ssh
send scp, sftp, rsync
reach ping
trace traceroute
lookup dig, nslookup, host

Archive and compression

new old
archive tar, cpio, zip via --create, --extract, --list
compress and expand gzip, bzip2, xz, zip, unzip

Miscellaneous

calc (and bc), echo, expr, false, seq, sleep, test, and true.

Examples

today becomes
ls -la list --long
du -sh * usage --human
ps aux processes
cat f | grep x | wc -l search x f | count
cut -d, -f1,3 select name, amount
sort -t, -k3 -n sort score --numeric
awk '$3 > 100 {print $1}' filter score > 100 | select name
grep -ri foo . search foo --recursive --ignore-case
find . -name '*.py' find . --name '*.py'
curl -s url fetch url
tar czf x.tgz dir archive dir --create --compress
sha256sum f checksum f --algorithm sha256
jq '.items[]' f read f | from json | flatten

Declaring a Schema

Every producer has a shared option --schema, which makes it emit a table that describes the fields it produces. list --schema, for example, returns:

field type description
name path file or directory name
type string file, directory, or symlink
size bytes size in bytes
modified timestamp time of last modification
mode string permission bits
owner string owning user
group string owning group

Because a schema is itself a table, the ordinary transformers and serializers apply to it: list --schema | select field, type | to json yields the compact machine-readable form, while list --schema alone renders it for a person. As a result, a reader can learn what a command will emit before running it, using the same vocabulary as any other data.

Further Reading

Intermediate Debugging Examples

I’m thinking about extending this set of debugging examples with the more complex examples described below. I’d be very grateful for feedback: would they be useful? Would they be useful to you? What other examples could be included, or should be included instead of some of the ones I’ve outlined?

Log Ingestion Pipeline

A command-line Python tool that reads application log files, normalizes each line, extracts structured fields, and loads them into SQLite for later querying.

Concurrent Web Crawler

A Python crawler using httpx with a thread or asyncio worker pool that fetches pages, normalizes URLs, deduplicates by content hash, and records pages and links in SQLite.

Library Lending API

A FastAPI application with SQLite persistence and an htmx front end for a small library: books, members, loans, and holds.

Job Queue with Retries

A Python worker pool backed by a SQLite jobs table that claims, runs, and marks jobs done, with retry and exponential backoff.

Checkout Wizard

A multi-step checkout workflow where an Alpine.js front end holds state across steps and a FastAPI backend validates and submits the order.

Report Generator

A Python command-line tool that reads SQLite, groups records into day/week/month buckets, and emits CSV or HTML.

Incremental Backup Tool

A Python command-line tool that walks a source tree, content-hashes files, deduplicates, and copies changed files into a backup directory while recording a manifest in SQLite.

Legacy Data Migration ETL

A Python script that reads a legacy SQLite schema and writes a normalized schema for the same data.

Notification Dispatcher

A Python daemon that polls a SQLite outbox table, sends each pending message (email or HTTP), and marks it sent.

Plugin-Based CLI

A Python command-line tool (argparse or typer) that loads plugins from a directory; plugins register subcommands and share a config object.

Metrics Dashboard

A dashboard where htmx polls a FastAPI endpoint for JSON and Alpine.js renders charts, backed by SQLite sample storage.

Session & Auth Service

A FastAPI application with SQLite and an htmx front end that logs users in, issues a session token, validates it in middleware, and invalidates it on logout.

LLM as Interface

I’m old enough to remember when graphical user interfaces appeared on desktop machines in the 1980s. Many of them were wrappers around pre-existing programs; while some programmers (including my younger self) sneered at them for being pointless eye candy, all of a sudden ordinary people didn’t have to remember a long list of obscure command-line spells in order to make computers do useful things.

I’m now old enough to remember when LLMs appeared in the 2020s. All of a sudden people didn’t have to learn how to write syntactically correct YAML or figure out which of Git’s eleventy jillion flags would safely and correctly do whatever they wanted to do. I’m now wondering whether “LLMs make it feasible for non-specialists who aren’t heavily invested to use X” is the same as “X’s interface never actually needed to be so user-hostile”, and whether “talk through it with an LLM” is going to make complex software more accessible in the same way that GUIs did.

Generating a Lesson

It took me the better part of a year to write Software Design by Example in Python. It would have had more impact and a wider readership if I’d done it a decade earlier, but I learned a lot along the way, and I’m proud of it.

It took less than ten minutes for DeepSeek to write this lesson explaining how a filesystem works. I told the LLM to read three of the existing lessons, then create one of similar length, depth, and style; it came back with working code and prose that’s as good as any first draft I might have created. I don’t know how well it would have done without my work to guide its pseudo-random number generation, but I expect I’m going to be thinking about this and its implications for quite some time.

What Else for the Sudonomicon?

I started work last year on an introduction to systems programming for data scientists. I assumed learners were comfortable with the material covered in the Carpentries workshops on the Unix shell and Python:

The topics I outlined then were:

Before I start fleshing out the existing material, I’d like to figure out what else ought to be included. Possibilities include:

I would be very grateful for guidance: what do you think a (data) scientist who’s starting to manage their own system needs to know?

Testing the Schelling Segregation Model

In 1971, economist Thomas Schelling asked whether mild individual preferences for similar neighbours could produce large-scale residential segregation. The answer is yes: even when each agent requires only 30% of occupied neighbours to share its type, the population self-organises into sharply segregated clusters. The Schelling model is an example of an agent-based model:

It’s a landmark result in computational social science because it shows that macro-level patterns can emerge from micro-level rules without central coordination. It’s fairly easy to implement in Python, but testing it is a challenge.

Implementation

The grid is an $N \times N$ array. Each cell holds one of three values: 0 (empty), 1 (red agent), or 2 (blue agent). The Moore neighborhood of a cell is the set of adjacent cells horizontally, vertically, or diagonally, and the same-neighbor fraction for an agent at position $(r, c)$ is:

$$f(r,c) = \frac{\text{occupied neighbors of the same type}}{\text{total occupied neighbors}}$$

An agent is satisfied when $f(r,c) \geq \theta$, where $\theta$ is the threshold. An agent with no occupied neighbors is treated as satisfied ($f = 1$) so that isolated agents do not drift endlessly across empty space.

We initialize the grid by placing equal numbers of red and blue agents at random in a fraction $(1 - p_e)$ of cells, where $p_e$ is the empty fraction. (Those empty cells provide the vacancies that dissatisfied agents move into.) In each simulation step, we do the following:

The satisfaction rate (i.e., the fraction of agents that meet the threshold) rises as the grid segregates, and reaches 1.0 when every agent is happy with its neighborhood. This happens pretty quickly:

Testing

So how do we test this? More specifically, what do we get when we ask an LLM to generate tests for this?

  1. Grid shape and contents
    • make_grid must produce a $(50 \times 50)$ array whose cells are all in ${0, 1, 2}$
  2. Balanced agent counts
    • Red and blue agents are built as equal halves of the occupied pool, so their counts differ by at most 1 (an off-by-one when the total number of agents is odd)
  3. Neighbor fractions
    • An isolated agent (no occupied neighbors) must return 1.0, matching the convention that isolated agents are considered satisfied
    • An agent surrounded entirely by the same type returns 1.0
    • One surrounded entirely by the opposite type returns 0.0
  4. Stable segregated grid
    • A $4 \times 4$ grid with red in the left two columns and blue in the right two columns is fully satisfied
    • Border agents have $5/8 = 0.625$ same-type neighbors which exceeds the threshold of 0.3
    • One step must leave the grid unchanged
  5. Satisfaction increases
    • Starting from the random initial grid, the satisfaction rate after 20 steps must exceed the initial rate because agents self-organise into clusters

The first three items don’t exercise the science at all. That doesn’t mean they’re not important, but they feel a bit trivial. The fourth and fifth items are more interesting, but take a close look at the fifth one. It is possible (though extremely unlikely) for a stochastic process to fail this test. How should we handle that? “Don’t worry about it” might actually be the right answer: as discussed last year, if the odds of a statistically-correct test failing are on par with the odds of two Git hashes colliding, we should turn our attention elsewhere.

The problem is that I don’t know how to calculate those odds. “It seems pretty unlikely to me” isn’t satisfying, particularly given how prone human beings are to confirmation bias. As more and more scientists generate code with LLMs, and use those same LLMs to generate tests, I’d be very grateful for answers to these questions.

Completed Fiction

I’ve written an average of about 450 words of fiction a day since I was laid off, but that number is misleading: some of those words have been rewritten several times. The more meaningful number is 28 rejections from agents and 25 from publishers in that same period.

And before anyone asks, no, I haven’t used AI to write any of this. Most of these projects are years old—I’m just taking advantage of my unexpected idleness to revise or rewrite them.