A Little Psychology

In order to understand how the world works, we have to understand how people think. That’s a tall order, so the sections below focus on a few things that I’ve found particularly useful.

See the first post in this series for context.

People Don’t Maximize Utility

In 1971, Daniel Kahneman and Amos Tversky ran a simple experiment. They told participants that a disease was expected to kill 600 people and asked them to choose between two public health programs. Program A would save exactly 200 people. Program B had a one-in-three chance of saving all 600 and a two-in-three chance of saving none. Most people chose A, i.e., they preferred the certain outcome.

Then Kahneman and Tversky rephrased the choice. Program C would result in exactly 400 deaths. Program D had a one-in-three chance that nobody would die and a two-in-three chance that all 600 would die. Statistically, the two programs are identical, but this time, most people chose D. Nothing changed except how the outcomes were described.

Classical economics assumes that people are rational agents who consistently maximize their own utility. When faced with a choice they weigh expected outcomes, discount the future at a consistent rate, and select whatever serves them best. This has been proven false over and over again, but persists because—well, we’ll get to that later in this essay.

Behavioral economics looks at how people actually make decisions, and has repeatedly shown that they deviate from “rational” in predictable ways. The first problem with the rational-agent model is computational. Optimizing requires evaluating all possible options against all possible outcomes under all possible conditions. No one can actually do this, so instead, people use a strategy that Herbert Simon called satisficing: they search through available options until they find one that is good enough and then stop. Herbert Simon called this bounded rationality: people are rational within the limits of the information, time, and cognitive capacity they actually have, which makes the heuristics people use to make decisions worth studying.

Kahneman and Tversky spent decades cataloguing people’s heuristics and the cognitive biases they embody. Anchoring is one of the most reliably reproduced findings in all of psychology. When people estimate an unknown quantity, their estimates are heavily influenced by numbers they have recently encountered, even ones they know to be irrelevant. In one study, participants spun a wheel rigged to land on either 10 or 65, then estimated the percentage of African countries in the United Nations. Those who had seen 65 guessed about 45 percent higher than those who had seen 10. They knew the wheel was random, but the number shaped their thinking anyway.

This isn’t stupidity or laziness; it is the brain doing something that is sensible in most contexts. Nearby numbers are usually informative, so most of the time, it makes sense to rely on them. This is why prosecutors set high anchor charges: juries’ verdicts cluster around the opening number. It is also why retailers display high “original” prices: customers anchor to whatever is crossed out. And research on salary negotiation consistently shows that the person who names the first number has the advantage, which is why negotiating advice boils down to the same instruction: speak first.

The availability heuristic says that people estimate how likely something is by how easily they can think of examples. After a plane crash receives extensive media coverage, people overestimate the risk of flying and underestimate the risk of driving, even though the underlying statistics have not changed. The availability heuristic is why catastrophic but rare events dominate public attention while slow, diffuse harms are systematically underestimated, which in turn is why it took decades to build public pressure around tobacco, lead paint, and vehicle safety [Kahneman2011].

Prospect theory describes how people actually evaluate outcomes. The key finding is loss aversion: a loss of a given size produces roughly twice the emotional impact of an equivalent gain. This asymmetry has practical consequences wherever people have a reference point they are trying to protect. Studies of taxi drivers in New York, Singapore, and other cities show that drivers work longer hours on bad days when earnings are below their daily target, and knock off early on good days. A rational agent who cares about total earnings would do the opposite, working more hours when conditions are favorable and fewer when they are not. Instead, drivers are managing losses relative to a reference point, not maximizing total income.

Similarly, the standard model predicts consistent discounting: a reward next month should be worth a fixed percentage less than the same reward today, and the same percentage should apply to any two adjacent future periods. What people actually show is hyperbolic discounting: an extremely steep preference for the present relative to any future point, combined with much flatter preferences among future periods. This is why someone can genuinely plan to quit smoking next year while lighting a cigarette. It is why gym memberships are purchased with full intention and then rarely used. Our future selves are strangers, and we are generous to ourselves and stingy with strangers.

If small changes in how choices are presented can have large effects on behavior, then choice architecture—the deliberate design of decision environments—is itself a policy tool. Thaler and Sunstein called one form of it nudging. The canonical example is pension enrollment. When workers must actively opt in to a pension plan, participation rates are typically around 50 to 60 percent. When workers are enrolled unless they actively opt out, participation rises to 80 to 90 percent, without any change to the financial terms. The UK government introduced automatic pension enrollment in 2012; by 2019, over ten million additional workers had joined workplace pensions as a direct result.

The UK’s Behavioural Insights Team, established in 2010, found that adding “Nine out of ten people in your area pay their taxes on time” to letters sent to late tax payers increased on-time payment rates by several percentage points. The intervention cost essentially nothing and recovered tens of millions of pounds in additional revenue. Nudges like thiat are not manipulation in the obvious sense—nothing is hidden and no options are removed. But the line between a nudge and a shove depends entirely on whose interests the design serves. Automatic enrollment in a pension plan serves the worker. Automatic enrollment in a subscription that is difficult to cancel serves the company. Variable reward schedules designed to maximize platform engagement are also nudges, built on the same science, serving a different master. Every infinite scroll, every notification badge, every “people who liked this also liked” recommendation is a behavioral economics intervention. The field that began by documenting human irrationality has become the primary toolkit for industrializing it [Thaler2009].

People Care About Fairness

Imagine you are given ten dollars to split with a stranger. You can offer them any amount you like. If they accept, you both keep your shares, but if they reject the offer, neither of you gets anything. A purely self-interested stranger, according to classical economics, should accept any positive offer—even one dollar—because one dollar is better than nothing.

When researchers ran this experiment across dozens of countries, they found that offers below thirty percent of the total were rejected roughly half the time: people would rather walk away with nothing than accept an outcome they perceived as unfair. In some communities, rejection rates were even higher.

This experiment, called the ultimatum game, has been run so many times and reproduced so reliably that its basic finding is no longer seriously contested. People care about fairness, punish violations of it at cost to themselves, and do so even with strangers they will never see again. This directly contradicts the assumptions that underlie most of modern economics, much conservative political thought, and a substantial proportion of technology design.

The idea that human beings are fundamentally self-interested did not emerge from evidence. It emerged from political argument, was later dressed in mathematical formalism, and eventually achieved the status of dogma. Thomas Hobbes, writing in 1651, described the natural condition of humanity as “a war of all against all.” Life without government, he argued, was “solitary, poor, nasty, brutish, and short.” Hobbes wasn’t reporting on anthropology—he was making a political case for sovereign authority. If humans are naturally predatory, then the powerful state he wanted is the only alternative to chaos.

Two centuries later, Herbert Spencer read Charles Darwin’s account of natural selection and announced that it confirmed what Hobbes had suspected. “Survival of the fittest” is Spencer’s phrase, not Darwin’s. Darwin described differential reproductive success; Spencer described a cosmic competition for dominance in which helping the weak was a biological error. Social Darwinism, as this cluster of ideas became known, provided intellectual cover for opposing labor rights and public health measures, and for fighting almost any intervention that might protect people from market outcomes. After all, if the weak lost, it was because Nature intended them to lose.

By the mid-twentieth century, economists had turned these self-serving rationalizations into mathematics. Homo economicus was a rational agent who consistently and accurately maximized his own utility. He did not care about fairness or make mistakes; he also didn’t care about other people’s welfare unless it affected his own. (And yes, I’m using the male pronoun deliberately.) The behavioral economics research described in an earlier post explains how this is fiction, but the model is not just wrong about how people think, but about what they want.

The ultimatum game is one of dozens of experiments that have been used to study human social preferences across cultures. Public goods games ask participants to contribute to a shared fund that pays out to everyone, including those who contribute nothing. Standard economic theory predicts that rational individuals will free-ride (i.e., contribute nothing while collecting their share of others’ contributions) until the fund collapses. In practice, initial contribution rates are typically between forty and sixty percent, and when participants can identify and punish free-riders, contribution rates rise and stay high.

People punish free-riders even when punishment costs them something. What’s more, they do it in one-shot interactions where there is no future reputation at stake. Samuel Bowles and Herbert Gintis spent years synthesizing this evidence, arguing that humans evolved not just as individuals competing for resources but as groups competing against other groups. Cooperation within groups enforced by altruistic punishment of defectors was a successful evolutionary strategy. The capacity for that cooperation, and the emotional responses that sustain it like fairness, shame, and indignation, are as deeply embedded in human nature as any appetite for self-interest [Bowles2011,Bregman2020].

None of this means humans are angels. Self-interested behavior is real. But so is cooperation, fairness, and punishment of norm violation. The question is which tendencies a given institutional design tends to elicit.

The Commons is Not a Tragedy

In 1968, the ecologist Garrett Hardin published an essay in which he described a common pasture open to all herders. Each one, acting rationally in their own interest, would add animals to the pasture until it was destroyed. The gains from each additional animal went to the individual herder, but the costs of the degraded pasture were borne by all. Self-interest would, inevitably, exhaust the commons. The only solutions Hardin could see were privatization or state regulation.

“The Tragedy of the Commons” became one of the most cited papers in academic history. It appeared in economics, political science, environmental policy, and law. Its intellectual framework shaped fisheries policy, water rights law, and debates about global climate governance [Hardin1968].

There was just one problem: Hardin hadn’t studied any actual commons. He had described an unmanaged commons with no rules, no governance, and no community. The historically managed commons of medieval England, the Alpine meadows of Switzerland, the forest communities of Japan, and the irrigation systems of Valencia and Bali all had elaborate rules developed over generations, mechanisms for monitoring compliance, graduated sanctions for violations, and processes for resolving disputes. They had been managing shared resources sustainably, in some cases, for centuries. The real tragedy in Hardin’s work was his ignorance of how the real world actually worked.

In contrast, the political scientist Elinor Ostrom spent her career studying actual systems. The picture that emerged was not a tragedy, but a sophisticated diversity of institutions, each one adapted to local conditions, and each one solving the collective action problem that Hardin had assumed was unsolvable without markets or states.

In Governing the Commons, Ostrom identified eight design principles that successful self-governing commons tend to share:

1. Members have clearly defined rights to the resource.
2. Rules are adapted to local conditions rather than imposed from outside.
3. People affected by the rules have meaningful input into changing them.
4. A system exists for monitoring both the resource and the behavior of users.
5. Sanctions are graduated—minor violations draw minor consequences.
6. Conflicts can be resolved quickly and cheaply.
7. External authorities recognize the community’s right to self-organize.
8. Larger systems are built from nested smaller ones.

In 2009, Ostrom was awarded the Nobel Prize in Economics for her work. The prize committee cited her demonstration that “economic analysis can shed light on most forms of social organization.” What her work actually demonstrated was narrower and more radical than that: that communities could govern shared resources sustainably without either privatizing them or handing them to the state, and that the dominant theoretical model had failed to predict this because it had assumed the wrong things about human nature.

If the evidence against homo economicus is this extensive, why does the model retain such a hold on policy and institutional design? Part of the answer is that the model is self-fulfilling in a useful way. If you design a system that assumes people will free-ride, you build in monitoring, penalties, and enforcement mechanisms. Those mechanisms signal distrust, which tends to erode the social norms that sustained voluntary cooperation. People who might have contributed voluntarily now respond to being treated as suspects. The system that assumed selfishness produces the selfishness it expected. In contrast, Ostrom’s communities worked partly because the institutions expressed trust: users had a voice in the rules, sanctions were proportionate rather than punitive, and the system treated people as members of a community rather than as threats to be managed.

Technology platforms have largely chosen the other path. Terms of service are written for adversaries. Moderation systems treat all users as potential bad actors. Engagement optimization assumes that appetites can be exploited. These choices reflect a theory of human nature, and that theory has consequences—not just for the products built on it, but for the kind of behavior those products elicit and reward.

Ostrom’s lesson is not that humans always cooperate. It is that cooperation is a realistic outcome if systems are designed to support it, and that assuming the worst tends to prevent the better from occurring. The tragedy of the commons was not inevitable. It was what happened when community governance was absent. Building that governance, it turns out, is something humans are rather good at, so long as institutions give them room to try [Ostrom2015].

People Care About Appearances

In 2001, the Norwegian government made its tax records publicly searchable online, so that every citizen could now look up what any other citizen earned. This was not entirely new—the country’s tax data had theoretically public for years—but the internet made it frictionless. Journalists could now scrub entire neighborhoods, neighbors could check each other out, and colleagues could compare their salaries with one another’s.

Ricardo Perez-Truglia used this moment as a natural experiment [PerezTruglia2020]. He tracked self-reported well-being before and after the records went online and found that the gap between higher- and lower-income Norwegians widened by 29%. Absolute incomes did not change; what did was knowing how you compared to other people.

This is the central finding of research on social standing: what people care about is not how much they have in absolute terms, but where they stand relative to those around them. It explains a long list of behaviors that seem irrational under standard economic assumptions.

Thorstein Veblen noticed this in 1899, before there were smartphones or social media (or economists to argue with his heresy). The Theory of the Leisure Class introduced the term conspicuous consumption to describe spending whose primary purpose is to signal social rank [Veblen1899]. His key insight is that the signal only works if it is costly: something that only the wealthy can afford communicates rank precisely because of its price.

Similarly, in a world where most people have to do physical labor, conspicuous leisure is only possible for the rich. As leisure became more broadly available, the signal shifted: today, being seen to be overworked and constantly in demand signals high status: the business traveler at the airport in the expensive suit checking email at midnight is the modern equivalent of the nineteenth-century aristocrat who demonstrably never lifted anything heavy.

Veblen pointed out that status competition is structurally self-defeating. If I buy a larger house to signal rank and my neighbors respond by buying larger houses, we have all spent money and all returned to the same relative position. The competition is real but the gains are illusory; the spending continues because the first person to stop stops loses ground to those who don’t.

Robert Frank built on Veblen’s work with a careful study of wage patterns within firms [Frank1985]. Standard economics predicts that workers will always move toward higher absolute pay: if they can earn more elsewhere, they will go elsewhere. Frank found that this prediction fails systematically. Workers at the bottom of a firm’s pay distribution are paid above their marginal productive value, while workers at the top are paid below it. The spread is not random: it is consistent with workers accepting lower total pay in exchange for higher rank within their peer group.

The implication is that a programmer who is the highest-paid person on a small team may prefer that position to being a lower-ranked member of a higher-paying team, even if the absolute salary differential favors the larger team. This is not irrationality: rank confers real benefits, so trading some income for rank is a sensible exchange. Standard economics fails to predict the trade only because it refuses to count rank as a good.

Frank’s local-rank argument helps explain the consistent finding in salary surveys that the highest correlate of worker satisfaction is not absolute pay but pay relative to colleagues doing similar work. Across many countries and industries, fairness within the reference group matters more than the number itself.

Fred Hirsch introduced the concept of positional goods, whose value depends on how many other people have it [Hirsch2015]. A house with an ocean view is a positional good: if everyone had a house with an ocean view, the view would cease to confer distinction. A senior job title, a degree from a prestigious school, or a table at an exclusive restaurant are all examples.

Hirsch pointed out that positional goods cannot be democratized. Refrigerators and mobile phones can eventually be afford by almost everyone, and everyone genuinely benefits. Positional goods cannot work this way. For example, if a prestigious university expands admissions to let in everyone who wants to attend, its value signal collapses. This is precisely what has happened with university degrees in wealthy countries since the 1960s. When only a small fraction of the population held degrees, a degree signaled something. As participation rates rose from 5 percent to 50 percent, the same degree began to signal much less, so the game shifted to which university, then to postgraduate qualifications, then to increasingly specific institutional prestige. Each generation has to spend more to achieve the same relative position as the previous one. This is not a problem that can be solved by making university cheaper or more accessible: that simply changes the positional good everyone is competing for.

The empirical case that rank rather than income drives well-being has been built up over two decades. An analysis of the British Household Panel Survey, which tracked thousands of households over many years, found that once income rank was included in the model, absolute income had no statistically significant effect on life satisfaction [Boyce2010]. What predicted whether someone was satisfied with their life was where they stood compared to their peers.

Wilkinson and Pickett extended this argument at the national level with evidence that more unequal societies perform worse on almost every social indicator, regardless of their average wealth [Wilkinson2011]. More equal societies have lower rates of homicide, mental illness, obesity, teenage pregnancy, and imprisonment. They have higher rates of trust, social mobility, and life expectancy. This pattern holds across wealthy countries: the United States, the United Kingdom, and Portugal, which are among the most unequal wealthy nations, perform poorly; Japan, the Nordic countries, and the Netherlands, which are among the most equal, perform well. The causal mechanism is status anxiety: higher inequality creates steeper hierarchies, which produce more corrosive competition for rank.

Which brings us to social media. Before digital platforms, status competition based primarily on physical proximity: you compared yourself to your neighbors, colleagues, and relatives. Platforms have replaced that bounded reference group with a global feed curated by algorithms optimized for engagement rather than accurate representation.

The comparison you are now offered is not with your actual neighbors. It is with the most aspirational version of everyone you have ever met. The result in South Korea, India, the UK, and Brazil is an intensification of status anxiety without any corresponding change in absolute circumstances. Someone whose life is objectively comfortable can be made to feel inadequate by a platform that continuously serves them evidence that other people are more attractive, a better parent, or has traveled more widely.

Social media platforms did not create the desire for status. What they did was put that desire on a subscription model, charge advertisers to place products in the resulting stream of anxiety, and call the resulting business a social network. It’s a game that only they can win.

Corporations are Psychopaths

In October 2011, Michael Woodford received what should have been the best news of his career. After thirty years working at Olympus, the Japanese optics and medical equipment company, he had been made chief executive: the first non-Japanese person to run the company in its history. Six months later, he was fired.

Woodford had found something wrong with a series of acquisitions the company had made. The amounts paid were enormous, the assets were nearly worthless, and the accounting explanations made no sense. He hired KPMG to look into it and took the resulting report to the chairman, Tsuyoshi Kikukawa. Kikukawa’s response was to call an emergency board meeting and vote Woodford out.

Woodford went public. The Olympus board denied everything for a few weeks, but then the numbers collapsed, Kikukawa resigned, and criminal charges followed. The fraud—a sustained effort to conceal $1.7 billion in losses—had been running for nearly twenty years, through the tenure of multiple CEOs. What is notable about the Olympus scandal is not that individuals behaved dishonestly. It is that they weren’t necessarily bad people. It was as if the organization had developed a mind of its own, and successive leaders served it rather than the other way around.

Three years before the Olympus scandal, the Canadian legal scholar Joel Bakan asked Robert Hare, the psychologist who had spent four decades developing the clinical tools used to diagnose psychopathy, to evaluate a publicly traded corporation against his Psychopathy Checklist as if the corporation were a person [Bakan2005]. The checklist was designed to identify individuals who are:

• callously indifferent to harm they cause to others,
• skilled at charming and manipulating people around them,
• incapable of genuine guilt or remorse,
• unwilling to accept responsibility for their own failures, and
• willing to lie and deceive when they believe they can get away with it.

Hare’s conclusion was that publicly traded corporations fit the profile.

In most jurisdictions, corporate executives have a fiduciary duty to shareholders: they are legally required to pursue shareholder interest, and a board that sacrificed profit to benefit workers or communities with no defensible business justification could be held legally liable. The resulting entity is therefore prohibited from having a conscience in the way an individual person might. It can behave ethically when ethics is good for the brand, but not when the cost cannot be justified by future returns.

None of this requires any individual inside the organization to be a bad person. It requires only that the rules governing the organization create incentives that produce a certain kind of behavior. However, this argument becomes harder to sustain when you look at who rises to the top of large organizations.

In 2005, Belinda Board and Katarina Fritzon surveyed 39 senior managers and executives in the United Kingdom and compared their psychological profiles to a matched group of patients at Broadmoor, a high-security psychiatric hospital [Board2005]. The executives scored higher than the Broadmoor sample on three personality disorder traits: histrionic, narcissistic, and compulsive. The researchers called this pattern “successful psychopathy”: the traits that lead to hospitalization or criminal conviction in their extreme form are, in a milder and better-managed form, associated with reaching senior management.

Paul Babiak and Robert Hare spent years studying how psychopathic individuals navigate organizational environments [Babiak2019]. Their estimate is that roughly one percent of the general population meets the clinical threshold for psychopathy, while corporate managers cluster around three to four times that rate. The mechanism is not mysterious. Psychopaths tend to perform exceptionally well in job interviews. They are confident, articulate, and skilled at saying what an interviewer wants to hear. They feel no social anxiety in high-stakes situations, and can fabricate credentials and relationships convincingly because they feel no guilt about doing so. Once hired, they are evaluated primarily on how they appear to those above them in the organization, and making a strong impression on a small number of people across a limited number of interactions is something psychopaths do better than almost anyone else.

This is where a closely related concept becomes important: impression management, a term introduced by Erving Goffman in 1959 [Goffman1959]. His observation was that social life is fundamentally theatrical: people perform different versions of themselves for different audiences, and success in social situations depends heavily on managing those performances. In a small organization where everyone works closely together over years, this has limited scope because your actual behavior is too visible. Colleagues know when you take credit for other people’s work, when your confident predictions turn out wrong, and when your charm disappears because you no longer need something from someone.

In a corporation with thousands of employees, on the other hand, promotions are typically decided by people who have less direct contact with the person in question. They evaluate based on presentations, meetings, secondhand reports, and the impressions formed in a relatively small number of interactions. This is precisely the environment where impression management skills are most valuable, and where the gap between managing impressions and actually performing well is hardest to detect.

Researchers who study the dark triad of psychopathy, narcissism, and Machiavellianism have consistently found that individuals high in these traits do particularly well in the early and middle stages of corporate careers. Narcissists project confidence and vision. Machiavellians are skilled at reading and exploiting organizational dynamics. Psychopaths can absorb stress, make decisions that harm others without losing sleep, and deliver bad news without visible discomfort. Each of these is a behavior that, in moderation and over short time horizons, looks like leadership. Failure only occurs when a crisis requires something the dark triad cannot supply: integrity, honest self-criticism, or concern for people the leader does not need.

Wirecard was a German payments company whose rise was celebrated as a European technology success story. By 2018 it had joined the DAX, Germany’s index of its thirty largest listed companies. Its chief executive, Markus Braun, appeared at industry conferences as the model of a visionary, unflappable founder. When the Financial Times published articles suggesting that large portions of the company’s claimed revenue did not exist, Germany’s financial regulator filed a criminal complaint against the journalist who wrote them. When the fraud collapsed in 2020, €1.9 billion turned out never to have existed. Braun had not built a company. He had built an extremely convincing impression of one.

South Korea’s chaebol are a structural variation on the same theme. The heads of Samsung, Lotte, SK, and others have faced criminal convictions for bribery and embezzlement—and received presidential pardons, typically on the grounds that their imprisonment would harm the national economy. This pattern of prosecution followed by pardon describes an organization that has achieved something psychopathic at the institutional level: the normal consequences of harmful behavior have been suspended because the organization is too important to be held accountable.

None of this means that every large company is led by psychopaths, or that organizational scale inevitably produces moral failure. It means that the selection pressures of large hierarchies are not neutral. Hiring processes that rely heavily on interviews systematically favor candidates who are good at interviews. Promotion decisions made by people with limited direct observation systematically favor candidates who are good at being observed, and performance reviews based on self-assessment systematically favor candidates who think highly of themselves. These processes aren’t designed to select for the dark triad, but they are all structured in ways that make dark triad traits an advantage.

The Olympus fraud ran for twenty years because each successive layer of management found it easier to maintain the deception than to stop it. No individual needed to be a psychopath; the organization’s incentives reproduced psychopathic behavior regardless of who ran it. Bad people come and go; structures that reward bad behavior reproduce themselves.

Why Don’t People Just Say No?

In 2016, Wells Fargo fired 5300 employees for opening millions of fake accounts in customers’ names without their knowledge. These were not executives: they were branch staff, customer service representatives, and personal bankers. When the scandal became public, initial coverage framed it as individual misconduct. The problem with that framing was that number “5300”. You cannot explain mass participation through individual bad character. Instead, you have to ask what conditions cause thousands of ordinary people to do something they know is wrong.

Stanley Milgram started asking this question in the early 1960s, partly in response to the trial of the Nazi Adolf Eichmann, who had organized the logistics of the Holocaust [Milgram1974]. Eichmann’s defense was that he had simply followed orders. Hannah Arendt, covering the trial for the New Yorker, coined the phrase that has not since been improved on: the “banality of evil.” Her point was that Eichmann was not a monster; he was a bureaucrat doing what his organization told him he was supposed to do [Arendt2006].

Milgram wanted to test how far ordinary people would go. In his experiments, volunteers were told they were measuring the effect of punishment on learning. An actor in another room pretended to receive electric shocks when giving wrong answers, and subjects were instructed to increase the voltage with each error. Most continued well past the point where the actor was screaming and, eventually, silent. Two thirds of subjects administered what they believed were the maximum possible shocks. (When the authority was absent or instructions were given by phone, compliance dropped sharply.)

Milgram’s subjects were not sadists: they were people responding to the combination of an authority figure, a legitimate-seeming purpose, and gradual escalation. Corporate hierarchies reproduce all of these conditions. The Wells Fargo employees were never actually instructed to defraud customers. They were given sales quotas that were mathematically impossible to meet through legitimate means, put under daily supervision, and subjected to a culture in which the phrase “eight is great” (i.e., eight accounts per customer) was a daily mantra. Each individual decision was small enough to feel manageable; employees who raised concerns were sometimes fired, and the outcome was fraud on a massive scale.

Albert Bandura spent decades studying what he called moral disengagement: the psychological mechanisms by which people participate in harmful behavior without experiencing it as harmful. These include displacement of responsibility (“I was just following orders”), diffusion of responsibility (“everyone else was doing it”), euphemistic labeling (calling fake accounts “cross-selling solutions”), and treating the people being harmed as abstractions rather than as individuals.

Bandura’s insight is that these mechanisms are not rationalizations invented after the fact [Bandura1999]. They are available in advance, and organizations learn to activate them. When tech companies describe user data as “exhaust”, call manipulative design patterns “engagement optimization”, or frame advertising surveillance as “connecting people with relevant products”, they are providing workers with the vocabulary of moral disengagement they need to get them to do morally repugnant things [Palazzo2025].

The pattern repeats across industries and cultures. Mitsubishi Motors covered up safety defects for over two decades, with participation from engineers, quality controllers, and managers who each knew parts of the problem. And then there’s Volkswagen’s “Dieselgate” scandal, which became public in 2015 after researchers at West Virginia University found that cars on the road emitted far more nitrogen oxides than official test results suggested [Ewing2017]. Engineers had written software that detected when a vehicle was undergoing an emissions test and activated pollution controls that were switched off during normal driving. Around eleven million cars worldwide contained this code, across multiple model lines and product generations. The engineers who wrote it had to design, maintain, and extend the software for years, which required knowing exactly what it was for.

What changes the equation? Milgram found that compliance fell dramatically when subjects could see another person refuse to continue. One dissenting voice—a confederate planted among the subjects—was enough to break the spell, which shows that the social proof of refusal is as powerful as the social proof of compliance. The implication is not that everyone needs to be a hero. It is that if you want ethical behavior from an organization, you only need a few visible dissenters.

Facts Alone Don’t Change Minds

Most scientists and programmers’ implicit model of belief is roughly Bayesian: when someone who believes something about the world receives new evidence, they update their beliefs in the way that fits that evidence best. This model is mostly true in domains that people aren’t emotionally invested in, but fails in predictable ways for beliefs that are tied to group membership.

Research in social psychology has established that beliefs about contested political and social issues function primarily as signals of group identity rather than as conclusions from evidence [SteinLubrano2024]. Holding the wrong belief does not just mean being misinformed: it puts you outside the group, so updating the belief means leaving that group. The social cost of updating is therefore often higher than the mental cost of staying wrong. This is why the tobacco industry’s manufactured uncertainty was so effective: it did not need to be persuasive on the merits. It only needed to give people with strong social reasons not to update a plausible excuse to stay where they were.

Motivated reasoning compounds this. People do not evaluate arguments neutrally. They are significantly better at identifying flaws in arguments that lead to conclusions they dislike than in arguments that lead to conclusions they support. A trained scientist who is also a gun owner will scrutinize studies on gun violence more skeptically than studies on climate change if their social circle treats the former as identity-threatening but not the latter. This isn’t dishonesty in the ordinary sense because the person doesn’t know they’re doing it. The biased scrutiny is real scrutiny; the flaws they find are often genuine. But the asymmetric attention means they have reached a conclusion before their evaluation begins [Kahneman2011].

A colleague once told me that people want data, but believe stories. This makes sense in light of motivated reasoning: data provides cover for a decision already reached on other grounds, while stories transmit the emotional and social context that actually drives belief change. It is also why the most effective public health campaigns don’t focus on presenting statistics; they present specific, named people in specific situations.

This rule is also why industries that want to prevent people from changing their beliefs fund think tanks that produce reports full of data. The data itself is not meant to influence people: instead, they count on the appearance of rigor, which mimics legitimate evidence without its substance.

The fossil fuel industry has been running this operation for thirty years, and tech companies are now doing it as well. If we want AI, social media, and the software industry in general to be regulated in meaningful ways, finding and presenting evidence of harm won’t be enough. We need to change the social context in which beliefs are held by finding trusted messengers within the relevant communities, reframing the issue so that updating does not require identity betrayal, and working through social networks rather than through arguments. This is not manipulation: it is taking the psychology of belief seriously [Achen2017,Hoffer2010].

This is part of Version 2 of this material. See the whole series or the bibliography, or email me with feedback.