More Psychology

OK, back to how people think…

See the first post in this series for context.

Why Do You Want What You Want?

On Easter Sunday 1929, a group of women walked down Fifth Avenue in New York City smoking cigarettes. They had been hired by Edward Bernays, a publicist working for the American Tobacco Company, to light up in public and treat their cigarettes as what Bernays called “torches of freedom.” Women smoking in public was a social taboo; framing the act as feminist defiance was designed to dissolve that taboo and open the female market to tobacco sales.

It worked.

Bernays was Sigmund Freud’s nephew, and he had taken his uncle’s ideas about unconscious desire and applied them to commerce. His insight was that you do not need to argue with people about whether they want something. You can create the conditions under which they will want it [Bernays2024].

The previous psychology post explained how people make decisions differently than the rational-actor model predicts. This post makes a related point: before you can ask how people choose, you need to ask where their preferences come from. Standard economics treats preferences as given: people arrive at markets with wants, and markets serve them. Thorstein Veblen was among the first to argue that this is not how consumption actually works. People don’t want things in the abstract; they want things based on their social position, relative to what the people around them have and display. The desire for a particular pair of shoes cannot be separated from the social meaning of those shoes in a specific time and place and to a particular peer group.

John Kenneth Galbraith pushed the argument further in The Affluent Society. He called it the “dependence effect”: the wants that production satisfies are themselves created by the process of production. Advertising does not serve existing desires: it manufactures new ones and attaches them to products. The economy doesn’t exist to satisfy people’s needs—it exists to perpetuate itself [Galbraith1998].

In South Korea, cosmetic surgery has grown into a multi-billion-dollar industry drawing patients from across East and Southeast Asia. The procedures most in demand—eyelid surgery, jaw reduction, and particular forms of rhinoplasty—track the standards of appearance disseminated through Korean entertainment products, which are themselves produced by a commercial industry with strong incentives to generate aspiration.

Digital platforms have scaled this dynamic. Algorithmic recommendation systems don’t just show you content that matches what you already want. They build a behavioral model from pauses, clicks, shares, and watch time, then serve content designed to maximize engagement. Your preferences at the end of an evening’s scrolling are partly an artifact of what the algorithm chose to show you. Recommendation systems are also preference-construction systems.

This is distinct from the nudging described in the previous psychology post. A nudge changes how a choice is presented without altering the options. What we’re describing here is shaping what you want before you arrive at a choice. In India, skin-lightening products generate annual revenues in the billions of dollars, sustained by advertising that associates lighter skin with success, desirability, and social mobility. The market for these products—many of which contain harmful ingredients—doesn’t reflect a natural preference. It reflects decades of advertising, colonial inheritance, and film industry imagery.

Markets can satisfy preferences efficiently. What they can’t do (or rather, what the people who profit from them won’t do) is tell you whether your preferences satisfying are worth having, who created them, or why [Packard2007,Veblen1899,Wu2016].

Fads and Bubbles

In February 1637, a single bulb of the Semper Augustus tulip sold in Amsterdam for the price of a house on a canal. If you had bought one of those bulbs in November 1636 and sold it in January 1637, you would have tripled your money. If you held it until March, you were ruined.

Tulip mania is the most entertaining entry in the catalog of financial bubbles because it involves flowers. But the mechanics are identical to what happened in 1720 with South Sea Company shares, in the 1840s with British railway stocks, in the 1980s with Japanese real estate, in 1999 with dot-com IPOs, in 2006 with Florida condos, and in 2021 with NFTs.

The first thing to understand about a bubble is that it is not entirely irrational while it is happening. Tulips were genuinely scarce and genuinely fashionable. Railway technology was genuinely transformative, and the internet did change commerce. Every bubble grows from a real phenomenon, which is precisely what makes it so effective at separating people from their money.

The second thing is the greater fool theory. You do not have to believe that a tulip bulb is worth a house. You only have to believe that someone else will pay you more for it next month than you paid for it today. This is rational behavior when prices are rising, but disastrous the moment they stop.

[Mackay1841] documented this pattern in 1841, which remains readable and depressing because every pattern he identified is still with us. Isaac Newton, one of the smartest people who ever lived, lost the equivalent of several million dollars in the South Sea Bubble of 1720. He reportedly said afterward that he could calculate the motions of heavenly bodies, but not the madness of men. He had already sold his shares at a profit, watched the price continue rising, bought back in near the peak, and then watched it collapse.

Robert Shiller, who won a Nobel Prize partly for studying this problem, argues that markets are driven less by rational calculation than by contagious stories [Shiller2015]. He calls this narrative economics: when a compelling story spreads through a population, it changes behavior, and changed behavior changes prices, which seems to confirm the story, so it spreads further. This feedback loop is difficult to interrupt because the people caught up in it are not obviously doing anything wrong. They are listening to their neighbors, watching prices, reading the news, and making bets based on the best available information. The information just happens to be mostly about what other people are betting.

The British Railway Mania of the 1840s combined technological innovation with genuine investment needs to produce something close to collective insanity. Between 1844 and 1846, Parliament approved over 400 new railway projects. Engineers, lawyers, and promoters collected fees regardless of whether the lines were ever built. Landowners extracted fortunes from right-of-way negotiations. Ordinary investors poured their savings into railways that had no surveyed routes, no locomotives, and sometimes no identifiable board of directors. When the bubble collapsed, it wiped out a significant fraction of the British middle class, destroyed several major banks, and left behind a network of railways that, once consolidated by the surviving companies, actually worked.

That last part matters. Bubbles are destructive for investors, but sometimes productive for infrastructure. The dot-com crash of 2000 that wiped out trillions of dollars in market value left behind fiber-optic cable in the ground, server farms, and the engineering knowledge to run large-scale internet services cheaply. Most of the companies that failed deserved to fail: their business models assumed that selling pet food online at a loss, while paying for national television advertising, would somehow eventually produce a profit. The crash was the market performing its nominal function of reallocating capital away from bad ideas. The problem was that the reallocation happened after several hundred thousand people lost their jobs and several million lost their retirement savings.

The housing bubble that collapsed in 2008 followed a different structure because it was built on debt rather than equity [Lewis2010]. Banks lent money to people who could not repay it, packaged those loans into securities that were sold to pension funds and insurance companies, bought insurance against those securities from companies that could not pay out, and collected fees at every stage of the transaction. The economists who designed the risk models assumed, based on historical data, that housing prices never declined nationally at the same time. They were right until they were catastrophically wrong.

Crucially, the losses did not fall on the banks that had made the bad loans—the US government covered those. The losses fell on homeowners who lost their houses and workers who lost their jobs when the broader economy contracted. John Kenneth Galbraith noted in his history of the 1929 crash that the capacity of human beings to ignore evidence that contradicts a profitable belief is essentially unlimited [Galbraith1954]. The 2008 crisis added the observation that when enough money is at stake, the strongest defenders of free markets suddenly become socialists in all but name.

Another feature of speculative bubbles is that they tend to be ahistorical. The dot-com bubble was the first internet bubble; there were no previous internet bubbles to learn from. The mortgage securities market of the 2000s was large enough that its collapse was qualitatively different from anything that had happened before. The phrase “this time it’s different” has become shorthand for the moment just before a collapse, but it is genuinely true in a narrow sense: the “why” of a particular collapse is usually novel enough that people who want to believe can point to real differences from previous disasters [Kindleberger2005].

Which brings us to AI. Large language models are genuinely impressive. The companies building them have produced tools that fundamentally change how people write and program. But the valuations of AI companies have, since roughly 2023, been running well ahead of their revenue, their profit margins, and any plausible estimate of the addressable market. The story driving those valuations is that AI will soon be able to do everything, which is a story that has been told before about nuclear power, and about previous generations of expert systems and neural networks.

This doesn’t mean that AI companies will collapse to nothing, any more than railways did. The real question is who is going capture the value and who is going to absorb the cost when reality catches up with the hype. Historically, the engineers and founders who cashed out early do well. The pension funds, retail investors, and laid-off workers who arrived later in the story do not, while the people who collected fees at every stage of the transaction do best of all.

The Myth of Meritocracy

The word “meritocracy” was coined by the British sociologist Michael Young in a satirical novel in 1958. Young’s dystopia describes a society that has perfected the measurement of individual talent and ruthlessly sorts people by it, producing a ruling class convinced of its own deservingness and contemptuous of those beneath it. The word was adopted by politicians and executives as a term of praise, and Young spent the rest of his life pointing out the confusion [Young1958].

How elite selection actually works in practice is very different from how institutions describe it. Studies of admissions to elite universities, hiring at professional services firms, and promotion within corporations consistently find that formal credentials matter less than social legibility: the ability to display the cultural capital like tastes, manners, and cultural references that signals membership in the right networks. Lauren Rivera’s research on hiring at elite firms found that interviewers routinely described candidates as “polished” or “rough around the edges” in ways that correlated with class background rather than competence. The stated meritocratic criteria were real, but they operated within a prior filter that most candidates never saw [Rivera2015,Karabel2006].

Audit studies make the gap between stated and actual criteria measurable. Researchers send identical resumes to employers, varying only the name at the top, and record callback rates. The results are consistent across countries and decades: resumes with names read as white receive significantly more callbacks than identical resumes with names read as Black or Latino. The same pattern appears when resumes signal class background or gender or when disability is disclosed.

Believing in meritocracy does not make people fairer. Research by Emilio Castilla and Stephen Benard found that organizations that explicitly adopt merit-based pay principles show larger gender pay gaps than those that do not. The mechanism appears to be moral licensing: people who believe the system is already fair feel less need to monitor their own judgments for bias. The belief in meritocracy also predicts harsher moral judgments of people who fail. After all, if outcomes are deserved, then failure is evidence of some personal deficiency. This framing conveniently shifts responsibility from structures to individuals and makes collective responses to poverty or unemployment harder to sustain politically [Sandel2020,Wilkinson2011].

The meritocratic ideology takes specific forms in the tech industry. The puzzle interview, popularized by Microsoft in the 1990s and still widespread, selects for people who have prepared for puzzle interviews, which correlates with having time, networks, and educational backgrounds that make such preparation possible. “Culture fit” as a hiring criterion is even less bounded: it is assessed subjectively and consistently reproduces the demographic composition of existing teams. Neither criterion is without any signal about likely job performance, but neither is calibrated against actual job performance data, because most organizations never close that loop.

The Representation Feedback Loop

In 1967, at the height of the civil rights movement, Nichelle Nichols played Lieutenant Uhura on Star Trek. She decided to quit after the first season, but Dr. Martin Luther King Jr. persuaded her to stay. He told her she was not playing a role but representing a future.

Whoopi Goldberg, having grown up without ever seeing a Black woman on television who was not playing a servant, ran through her house shouting when she first saw Nichols on screen. Mae Jemison, the first Black American woman to travel to space, has said that watching Uhura was formative for her sense of what was possible. These stories and millions of others show that representation is not just symbolic: it changed what futures became thinkable.

Research has shown that men outnumber women roughly three to one in speaking roles in family-rated films, and that the gap is even larger among characters in science and technology careers. If every doctor, lawyer, and CEO you ever see looks a particular way, then looking like them becomes part of your picture of what belonging in those roles requires. This is not a matter of being naïve or impressionable; it is just how our minds work.

Breaking the cycle requires intervening somewhere in it, which is precisely where resistance concentrates. The NFL’s Rooney Rule, introduced in 2003, required teams to interview at least one minority candidate for head coaching positions. The argument against it was that it was condescending—that qualified candidates should be hired on merit, not to satisfy a quota. The argument for it was that the existing process was already not meritocratic: the informal networks through which coaching positions were filled were built from relationships among people who looked the same, so identical qualifications led to different outcomes depending on whose network you were in. The Rooney Rule did not guarantee hiring, but even that minimal intervention increased the number of minority head coaches appointed in the years following its introduction.

Algorithmic recommendation systems reproduce the same dynamic at scale. YouTube’s algorithm surfaces content similar to what previously earned engagement, which means creators from underrepresented groups face higher barriers to discovery than their more visible counterparts. The algorithm is not making a judgment about quality: it is applying a preference filter built from historical patterns, and those patterns reflect who had access to platforms and production tools before the algorithm existed.

This framing is what makes the cycle self-perpetuating. If you believe that whoever is currently represented in any field is there because they were the most qualified, then any change in the composition looks like a lowering of standards rather than a correction of a filter. The response to evidence of the filter is to explain that the filter must be working correctly, because look at the results. The loop closes there unless something breaks it open.

Why Discrimination Persists

Becker’s 1957 work on discrimination in labor markets generated an optimistic prediction [Becker1971]. Firms that discriminate pay a cost because they forgo productive workers that non-discriminating competitors will hire instead. Over time, competitive pressure should therefore eliminate discrimination.

History has thoroughly falsified this prediction. Discrimination has not been competed away in any labor market where economists have looked carefully. The gap between Becker’s prediction and the observed reality reveals that rational economics misidentifies what discrimination is and how it works.

Economists distinguish taste-based discrimination, in which employers simply prefer not to hire members of certain groups regardless of productivity, from statistical discrimination, in which employers use group membership as a proxy for characteristics they cannot directly observe. An employer who uses race or gender as a proxy for something like reliability is drawing inferences from population-level patterns that are themselves often the product of prior discrimination, which creates a vicious circle.

The psychological research on implicit bias complicates the picture further [Banaji2013]. Studies using the Implicit Association Test and its successors show that most people, including those who explicitly reject prejudice, harbor automatic associations between social groups and evaluative attributes. These associations operate below the conscious level, and lead to well-intentioned people evaluating resumes differently depending on the apparent race or gender of the applicant.

Audit studies provide the cleanest evidence. Researchers send pairs of fictitious resumes to real job postings, holding everything constant except for names that signal racial identity. The results are consistent across many countries and many decades: resumes with names coded as white receive significantly more callbacks than identical resumes with names coded as Black. The same pattern appears when resumes signal class background, gender, or disability.

But understanding why discrimination persists also requires recognizing that some workers benefit from it. Workers who belong to favored groups face less competition for jobs, promotions, and wages when members of excluded groups are kept out. This gives them an incentive to back political groups that will maintain discriminatory arrangements, even if they object adamantly to being accused of discrimination [McGhee2021]. Such groups merely need people to believe that not actively being bad is the same as being good.

India’s caste system and South Africa’s apartheid both persisted long after any plausible economic case for them had dissolved. They persisted because they served social and psychological functions that economic analysis does not capture: in particular, they guaranteed those near the bottom of otherwise low-status hierarchies that they were not at the very bottom.

If you can convince the lowest white man he’s better than the best colored man, he won’t notice you’re picking his pocket. Hell, give him somebody to look down on, and he’ll empty his pockets for you.

– Lyndon B. Johnson

The history of how discrimination gets built into professional structures is worth understanding before applying it to computing. The Flexner Report of 1910 recommended consolidating medical education around a small number of university-based schools modeled on Johns Hopkins. What is less often noted is what this consolidation closed: the eclectic and homeopathic schools that had trained a significant proportion of women physicians, and the historically Black medical colleges that were the primary pathway into medicine for Black practitioners. The reorganization of medicine around a single credentialing model was also a reorganization of medicine around a single demographic profile. The American Medical Association’s subsequent decades of opposition to women’s participation in medical societies and to Black physicians’ hospital admitting privileges extended and consolidated what the Flexner Report had begun.

The legal profession followed the same pattern. Bar associations across Britain, the United States, and continental Europe formally prohibited women from practice until courts or legislatures intervened. In the UK, women were not admitted to the legal profession until 1919. In the US, the Supreme Court upheld Illinois’s exclusion of women from the bar in 1872, with a concurrence that stated women’s domestic roles were divinely ordained. The formal barriers were eventually removed. The informal barriers persisted: partnerships that did not hire women, chambers that did not take women as pupils, and professional associations that did not see women as natural members. These required decades more of pressure to dislodge, and have not been fully dislodged.

The undercounting of women’s scientific contributions is documented across many countries and disciplines. The pattern includes attribution of women’s work to male collaborators or supervisors, systematic exclusion from authorship norms that credited the laboratory head rather than the person who performed the experiments, and deliberate suppression documented in specific cases such as Rosalind Franklin’s role in the determination of DNA’s structure. Studies of citation patterns, grant success rates, and peer review outcomes continue to find systematic disadvantages for women researchers in many scientific fields, controlling for measurable differences in output.

Mary Beard’s analysis of women’s exclusion from public speech draws the pattern to its full historical length [Beard2017]. From the moment in the Odyssey where Telemachus tells Penelope to go back inside and let men deal with public affairs, through Roman rhetoric’s treatment of women’s public speech as inherently disgraceful, through centuries of legal and customary prohibition on women speaking in court and in public assemblies, the mechanisms of exclusion are remarkably consistent. They include treating women’s speech as categorically out of place in authoritative contexts, dismissing or ridiculing women who do speak in those contexts rather than engaging with what they say, and redefining the terms of authority in response to women’s gains so that what women achieve is always reclassified as less important than what they have not yet achieved.

In contemporary computing, the primary mechanism of exclusion operates under the label of culture fit. Hiring decisions made on the basis of cultural fit are rarely experienced as discriminatory by those who make them. The judgment is framed as a question of whether a candidate seems like one of us, whether they would enjoy working with the existing team, whether they share the team’s values and habits. But when the existing team is demographically homogeneous, hiring for cultural fit reproduces that homogeneity. The criterion functions as a demographic filter while providing its users with a clean conscience.

The automation of hiring decisions does not solve this problem. It replicates it at scale. Algorithmic hiring tools trained on historical hiring data learn to identify candidates who resemble those previously hired and promoted. If the historical data reflects discriminatory patterns, the algorithm encodes those patterns into a system that is harder to scrutinize and challenge than a human decision-maker. Amazon’s internal recruiting tool, abandoned in 2018, penalized resumes that included the word “women’s” and downgraded graduates of all-women’s colleges, because it had learned from a decade of prior hiring decisions that were heavily male.

The standard response to demographic imbalance in tech is to point to the pipeline: not enough women or underrepresented minorities in computer science programs, and the problem will correct itself when universities produce more qualified candidates. This explanation is flatly contradicted by the evidence. Women’s participation in computing was substantially higher in the 1980s than it is today. The decline in women’s representation occurred while women’s representation in law, medicine, and other professions was increasing. The pipeline analogy locates the problem in the wrong place. Demographic problems that originate in hiring practices, retention conditions, and professional culture cannot be solved by changing who enters the pipeline. They require changing the conditions that determine who stays in, who is promoted, and who leaves.

The Moral Psychology of Building Harmful Things

In September and October 2021, the Wall Street Journal published a series of articles based on tens of thousands of internal Facebook documents provided by Frances Haugen. Among other things, the documents showed that Facebook’s own researchers had found that Instagram worsened body image issues and increased suicidal ideation in teenage girls, and that the company had known this for years while publicly denying that its platforms caused harm. The documents also showed that Facebook’s algorithms systematically promoted outrage and divisive content because it generated more engagement, and that internal teams had identified this as a serious problem without being able to change it.

None of this is surprising in retrospect. What is surprising is how ordinary the behavior looks when you read the documents: working groups, slide decks, and recommendations sent to committees. The harm was not produced by villains. It was produced by an organization operating in ways that feel completely familiar.

The psychologist Stanley Milgram ran his obedience experiments at Yale in 1961 and 1962, partly in response to the trial of Adolf Eichmann, who had organized the logistics of the Holocaust and whose defense was that he had simply followed orders [Milgram1974]. In the most famous version of Milgram’s experiments, subjects were told they were participating in a study of learning and were instructed by an authority figure to administer electric shocks to another person whenever that person gave a wrong answer. The other person was a confederate; the shocks were fake; the screams were recorded. Sixty-five percent of subjects administered what they had been told was the maximum voltage — labeled “Danger: Severe Shock” — because a person in a lab coat told them to. Milgram’s explanation was that people in hierarchical situations shift into an agentic state: they stop regulating their own behavior against their personal moral standards and start executing instructions from whoever they perceive as legitimate authority, locating responsibility upward rather than in themselves.

The psychologist Albert Bandura identified eight mechanisms by which people disengage their moral standards to engage in or tolerate harmful behavior without feeling responsible for it. Moral disengagement does not require cruelty or indifference. It operates through cognitive reframing available to people who consider themselves good. Moral justification frames the harm as serving a higher purpose, while euphemistic labeling describes content moderation failures as “trust and safety challenges.” Displacement of responsibility points to someone else: the advertisers demanded this, the users clicked on it. Diffusion of responsibility distributes it across so many people that no single one feels it: the engineers wrote the algorithm, the product managers set the metrics, the executives approved the strategy. Treating teenage users primarily as engagement statistics accomplishes much of the same work as dehumanization [Bandura1999,Palazzo2025].

The phrase “we just build hammers” is the most common form of moral disengagement in technology work. It frames engineers as neutral suppliers of capability, placing all responsibility for consequences on whoever chooses to use the tools. This framing collapses when you look at what the tools actually do. A recommendation algorithm that is trained to maximize engagement and deployed on a platform used by teenagers is not neutral the way that a hammer is. It is a system specifically designed to capture attention, running on knowledge about how human psychology responds to certain kinds of content, at a scale that no individual user can resist or comprehend. The programmers who built it made choices about what to optimize, and their bosses approved those choices.

None of this means that people inside organizations that cause harm are helpless. It means that the conditions that produce harmful outcomes are predictable, that they operate through ordinary human psychology rather than through exceptional malice, and that understanding them is prerequisite to changing them. When the metrics that determine whether a career succeeds are the same metrics that produce harm, most people will find ways to make peace with the harm [Ehmke2025].

This is part of Version 2 of this material. See the whole series or the bibliography, or email me with feedback.