The Undoing Project: A Friendship that Changed the World(45)

The mean IQ of the population of eighth graders in a city is known to be 100. You have selected a random sample of 50 children for a study of educational achievement. The first child tested has an IQ of 150. What do you expect the mean IQ to be for the whole sample?

At the end of the summer of 1969, Amos took Danny’s questions to the annual meeting of the American Psychological Association, in Washington, DC, and then on to a conference of mathematical psychologists. There he gave the test to roomfuls of people whose careers required fluency in statistics. Two of the test takers had written statistics textbooks. Amos then collected the completed tests and flew home with them to Jerusalem.

There he and Danny sat down to write together for the first time. Their offices were tiny, so they worked in a small seminar room. Amos didn’t know how to type, and Danny didn’t particularly want to, so they sat with notepads. They went over each sentence time and again and wrote, at most, a paragraph or two each day. “I had this sense of realization: Ah, this is not going to be the usual thing, this is going to be something else,” said Danny. “Because it was funny.”

When Danny looked back on that time, what he recalled mainly was the laughter—what people outside heard from the seminar room. “I have the image of balancing precariously on the back legs of a chair and laughing so hard I nearly fell backwards.” The laughter might have sounded a bit louder when the joke had come from Amos, but that was only because Amos had a habit of laughing at his own jokes. (“He was so funny that it was okay he was laughing at his own jokes.”) In Amos’s company Danny felt funny, too—and he’d never felt that way before. In Danny’s company Amos, too, became a different person: uncritical. Or, at least, uncritical of whatever came from Danny. He didn’t even poke fun in jest. He enabled Danny to feel, in a way he hadn’t before, confident. Maybe for the first time in his life Danny was playing offense. “Amos did not write in a defensive crouch,” he said. “There was something liberating about the arrogance—it was extremely rewarding to feel like Amos, smarter than almost everyone.” The finished paper dripped with Amos’s self-assurance, beginning with the title he had put on it: “Belief in the Law of Small Numbers.” And yet the collaboration was so complete that neither of them felt comfortable taking the credit as the lead author; to decide whose name would appear first, they flipped a coin. Amos won.

“Belief in the Law of Small Numbers” teased out the implications of a single mental error that people commonly made—even when those people were trained statisticians. People mistook even a very small part of a thing for the whole. Even statisticians tended to leap to conclusions from inconclusively small amounts of evidence. They did this, Amos and Danny argued, because they believed—even if they did not acknowledge the belief—that any given sample of a large population was more representative of that population than it actually was.

The power of the belief could be seen in the way people thought of totally random patterns—like, say, those created by a flipped coin. People knew that a flipped coin was equally likely to come up heads as it was tails. But they also thought that the tendency for a coin flipped a great many times to land on heads half the time would express itself if it were flipped only a few times—an error known as “the gambler’s fallacy.” People seemed to believe that if a flipped coin landed on heads a few times in a row it was more likely, on the next flip, to land on tails—as if the coin itself could even things out. “Even the fairest coin, however, given the limitations of its memory and moral sense, cannot be as fair as the gambler expects it to be,” they wrote. In an academic journal that line counted as a splendid joke.

They then went on to show that trained scientists—experimental psychologists—were prone to the same mental error. For instance, the psychologists who were asked to guess the mean IQ of the sample of kids, in which the first kid was found to have an IQ of 150, often guessed that it was 100, or the mean of the larger population of eight graders. They assumed that the kid with the high IQ was an outlier who would be offset by an outlier with an extremely low IQ—that every heads would be followed by a tails. But the correct answer—as produced by Bayes’s theorem—was 101.

Even people trained in statistics and probability theory failed to intuit how much more variable a small sample could be than the general population—and that the smaller the sample, the lower the likelihood that it would mirror the broader population. They assumed that the sample would correct itself until it mirrored the population from which it was drawn. In very large populations, the law of large numbers did indeed guarantee this result. If you flipped a coin a thousand times, you were more likely to end up with heads or tails roughly half the time than if you flipped it ten times. For some reason human beings did not see it that way. “People’s intuitions about random sampling appear to satisfy the law of small numbers, which asserts that the law of large numbers applies to small numbers as well,” Danny and Amos wrote.

This failure of human intuition had all sorts of implications for how people moved through the world, and rendered judgments and made decisions, but Danny and Amos’s paper—eventually published in the Psychological Bulletin—dwelled on its consequences for social science. Social science experiments usually involved taking some small sample from a large population and testing some theory on it. Say a psychologist thought that he had discovered a connection: Children who preferred to sleep alone on camping trips were somewhat less likely to participate in social activities than were children who preferred eight-person tents. The psychologist had tested a group of twenty kids, and they confirmed his hypothesis. Not every child who wanted to sleep alone was asocial, and not every child who longed for an eight-person tent was highly sociable—but the pattern existed. The psychologist, being a conscientious scientist, selects a second sample of kids—to see if he can replicate this finding. But because he has misjudged how large the sample needs to be if it is to stand a good chance of reflecting the entire population, he is at the mercy of luck.* Given the inherent variability of the small sample, the kids in his second sample might be unrepresentative, not at all like most children. And yet he treated them as if they had the power to confirm or refute his hypothesis.