Manufacturing Obedience

In 2016, Wells Fargo fired 5300 employees for opening millions of fake accounts in customers’ names without their knowledge. These were not executives: they were branch staff, customer service representatives, and personal bankers. When the scandal became public, initial coverage framed it as individual misconduct. The problem with that framing was that number “5300”. You cannot explain mass participation through individual bad character. Instead, you have to ask what conditions cause thousands of ordinary people to do something they know is wrong.

Stanley Milgram started asking this question in the early 1960s, partly in response to the trial of the Nazi Adolf Eichmann, who had organized the logistics of the Holocaust. Eichmann’s defense was that he had simply followed orders. Hannah Arendt, covering the trial for the New Yorker, coined the phrase that has not since been improved on: the “banality of evil.” Her point was that Eichmann was not a monster; he was a bureaucrat doing what his organization told him he was supposed to do.

Milgram wanted to test how far ordinary people would go. In his experiments, volunteers were told they were measuring the effect of punishment on learning. An actor in another room pretended to receive electric shocks when giving wrong answers, and subjects were instructed to increase the voltage with each error. Most continued well past the point where the actor was screaming and, eventually, silent. Two thirds of subjects administered what they believed were the maximum possible shocks. (When the authority was absent or instructions were given by phone, compliance dropped sharply.)

Milgram’s subjects were not sadists: they were people responding to the combination of an authority figure, a legitimate-seeming purpose, and gradual escalation. Corporate hierarchies reproduce all of these conditions. The Wells Fargo employees were never actually instructed to defraud customers. They were given sales quotas that were mathematically impossible to meet through legitimate means, put under daily supervision, and subjected to a culture in which the phrase “eight is great” (i.e., eight accounts per customer) was a daily mantra. Each individual decision was small enough to feel manageable; employees who raised concerns were sometimes fired, and the outcome was fraud on a massive scale.

Albert Bandura spent decades studying what he called moral disengagement: the psychological mechanisms by which people participate in harmful behavior without experiencing it as harmful. These include displacement of responsibility (“I was just following orders”), diffusion of responsibility (“everyone else was doing it”), euphemistic labeling (calling fake accounts “cross-selling solutions”), and treating the people being harmed as abstractions rather than as individuals.

Bandura’s insight is that these mechanisms are not rationalizations invented after the fact. They are available in advance, and organizations learn to activate them. When tech companies describe user data as “exhaust”, call manipulative design patterns “engagement optimization”, or frame advertising surveillance as “connecting people with relevant products”, they are providing workers with the vocabulary of moral disengagement they need to get them to do morally repugnant things.

The pattern repeats across industries and cultures. Mitsubishi Motors covered up safety defects for over two decades, with participation from engineers, quality controllers, and managers who each knew parts of the problem. And then there’s Volkswagen’s “Dieselgate” scandal, which became public in 2015 after researchers at West Virginia University found that cars on the road emitted far more nitrogen oxides than official test results suggested. Engineers had written software that detected when a vehicle was undergoing an emissions test and activated pollution controls that were switched off during normal driving. Around eleven million cars worldwide contained this code, across multiple model lines and product generations. The engineers who wrote it had to design, maintain, and extend the software for years, which required knowing exactly what it was for.

What changes the equation? Milgram found that compliance fell dramatically when subjects could see another person refuse to continue. One dissenting voice—a confederate planted among the subjects—was enough to break the spell, which shows that the social proof of refusal is as powerful as the social proof of compliance. The implication is not that everyone needs to be a hero. It is that if you want ethical behavior from an organization, you only need a few visible dissenters.

see the whole series · email me

Arendt2006

Hannah Arendt: Eichmann in Jerusalem: A Report on the Banality of Evil. Penguin, 2006, 9780143039884.

Bandura1999

Albert Bandura: “Moral Disengagement in the Perpetration of Inhumanities.” Personality and Social Psychology Review, 3(3), 1999, 10.1207/s15327957pspr0303_3.

Ewing2017

Jack Ewing: Faster, Higher, Farther: The Volkswagen Scandal. W. W. Norton, 2017, 9780393254501.

Frenkel2021

Sheera Frenkel and Cecilia Kang: An Ugly Truth: Inside Facebook’s Battle for Domination. Harper, 2021, 9780062960672.

Milgram1974

Stanley Milgram: Obedience to Authority: An Experimental View. Harper & Row, 1974, 9780061312946.

Palazzo2025

Guido Palazzo and Ulrich Hoffrage: The Dark Pattern: The Hidden Dynamics of Corporate Scandals. PublicAffairs, 2025, 9781541705302.