This book explores the concept of noise, an error of judgment that affects human decisions. The authors—Daniel Kahneman, Olivier Sibony, and Cass Sunstein—explain what is noise, how it affects decisions at all levels, and how we can mitigate noise to improve the efficiency, consistency and fairness of our decisions and systems. The ideas are relevant for decision-makers in all walks of life (e.g. executives, professionals, policy-makers), and anyone who’s interested in learning about psychology, behavioral economics, and organizational behavior. In this Noise summary, you’ll get an overview of noise and its implications.
What is Noise and Why It Matters
Human judgments (e.g. forecasts, performance evaluations, forensics analyses) are inconsistent and inaccurate. Given the same context and information, different people will make different judgments or decisions. Such inconsistencies come from noise and bias.
NOISE VS BIAS
Noise refers to the random variability and inconsistencies in judgment which lead people to evaluate the same information differently. It includes:
• In-person variability which occurs when the same person judges something differently under different circumstances.
• Between-person variability which occurs when different people judge the same case differently.
For example, 5 managers may evaluate the same business proposal differently. The same manager may even evaluate the same proposal differently on different days depending on his/her mood or focus.
Biases refer to the systematic errors in thinking that consistently skew judgments in a certain direction. For example, all 5 managers in a company may evaluate a business proposal too optimistically because of an optimism bias.
Both noise and bias contribute to flawed decision-making. Yet, relatively little attention has been paid to noise compared to biases.
In this free Noise summary, you’ll get learn about noise and its implications, with an overview of the solutions for noise. For a deep-dive into these ideas, complete with examples and details tips about noise, do check out our full 17-page Noise summary bundle including infographic, text and audio summaries.
Understanding Noise
NOISE EXISTS EVERYWHERE
Noise exists in any situation that involves human judgments—from legal systems to medicine, financial markets, and hiring practices. For example, we expect the criminal justice system to be fair and consistent, when it’s actually very noisy. A criminal can get a drastically different sentence depending who’s the judge presiding over the case, or even the judge’s mood that day. In the USA, the prison term for extortion ranges from 3-20 years, while the term for fraud ranges from 8.5 years to life imprisonment.
Noise exists even if it can’t be detected or measured.
Recurrent decisions are made repeatedly, such as underwriters quoting insurance premiums based on their clients’ health situation. In such cases, you can identify the extent/impact of noise though a “noise audit”. An insurance company was able to measure the variance in quotations by assigning 5 cases to different groups of underwriters. They found a median difference of 55%, which meant that the same client may be quoted $9,500 or $16,700 depending on which underwriter was assigned to the case. The variations were estimated to cost the company hundreds of millions of dollars. If the quotes are too high, the company loses a potential client. If the quotes are too low, the company loses money from large amounts of insurance claims.
Singular decisions are one-off decisions for unique problems. We can’t measure noise in such cases, e.g. you cannot compare 2 presidents’ decisions to go to war because the circumstances are different. Yet, the same factors that cause noise in recurrent decisions also exist in singular decisions. So, it stands to reason that noise exists in singular decisions even if you can’t measure it. The strategies that reduce noise in recurrent decisions should also improve the quality of singular decisions.
System noise is a problem.
Variability isn’t always bad. For instance, it’s fine for people to have different opinions about movies or wine. It’s also helpful to explore diverse viewpoints for complex issues like economic forecasts or developing a new vaccine.
However, system noise is always unwanted because the goal of judgments is to be accurate, not to express personal preferences. You don’t want inconsistencies in areas like criminal sentences, product/service quality, or employee performance. For example, you don’t want a different sentence depending on the judge for a court case, nor a different diagnosis depending on the doctor you consult. Yet, noise is tolerated because we often don’t even realize that it exists.
EVALUATING JUDGMENTS & MEASURING NOISE
How to Evaluate Judgments
In our complete Noise summary, we’ll explain more about:
• Predictive vs evaluative judgments
• Verifiable vs non-verifiable judgments
• Evaluating the quality of predictions based on outcomes vs processes
• Measures used in predictions: percent concordant (PC) and correlation coefficient (r)
How to Measure Noise
For now, we’ll jump straight into the best way to measure noise : using standard deviation, which shows how much a judgment has deviated from the mean value.
Overall Error (MSE) = Bias2 + Noise2
A higher MSE means greater variability or inconsistency, whereas a lower MSE means greater accuracy or consistency. Reducing either noise or bias will reduce the overall error in judgment.
• In noisy systems, errors add up; they don’t cancel out one another. For example, an overpriced insurance premium doesn’t compensate for an underpriced premium since they are for different clients.
What are the Components of System Noise?
There are 2 main components of system noise: Level Noise and Pattern Noise.
Level noise occurs when a 1 person’s judgments consistently differ from the average person’s, e.g. some judges are generally more lenient than others.
Pattern noise occurs when 1 person is consistently affected by specific factors, e.g. a judge is generally more lenient toward older criminals. It includes:
(i) Stable pattern noise: which comes from our internal beliefs, biases, values, or knowledge/experiences. We may weigh some factors more heavily than others, or react positively/ negatively to specific cues. Such patterns are relatively stable, though they’re hard to predict.
(ii) Occasion noise: which refers to random errors that affect your judgment, such as your mood, weather, state of mind, or how the information is presented. For example, one study found that college admissions officers weighed academic credentials more heavily on cloudy days, and weighed non-academic factors more heavily on sunny days. Occasion noise can only be measured with large-enough samples that statistically show the effects of occasion-related factors.
Where Does Noise Come From?
The components of noise (level noise, pattern noise, occasion noise) can be understood from a psychological perspective.
In our complete Noise summary, we’ll be zooming in on each of these sources of noise with specific examples. These include; mental heuristics, confusing causation vs correlation, differences in how we use/inteprete scales, overconfidence, and how groups can amplify noise.
For example, in the book Thinking, Fast and Slow, Daniel Kahneman explains how the brain uses mental shortcuts or heuristics to think rapidly and intuitively. However, this leads to psychological biases, including the substitution bias, conclusion bias, and halo effect. Psychological biases reduce our quality of judgments because they create statistical bias when the biases are widely shared, as well as system noise when individuals show different types/levels of biases. Another mental shortcut is “matching”, which is elaborated further in our full book summary of Noise.
How to Reduce Noise?
Noise Reduction Strategies
Noise-reduction strategies can be applied to improve the accuracy of all types of judgments. These include:
• Doing a noise audit to understand the extent and impact of noise in your system;
• Choosing good judges with the intelligence, experience/expertise and the right mindset (actively open-minded);
• Using de-biasing to counter psychological biases; and
• Adopting decision hygiene strategies, which are basically prevention strategies like washing your hands to prevent the spread of germs. Since noise is hard to see or predict, decision hygiene is the best way to reduce noise. The authors cover a wide range of decision hygiene strategies in the book, including: how to avoid ambiguous terms/scales, use relative judgments, aggregate large amounts of data and independent forecasts, and use mechanical predictions and algorithms. They also explain how to combine various decision hygiene strategies into the “Mediating Assessment Protocol.”
Noise Reduction in Real Life
The noise-reduction strategies above can be used to improve all types of judgments, from sales forecasts to medical diagnoses. Here is just 1 example.
The employee hiring process is meant to help organizations select candidates who’re likely to perform well in their roles. Yet, most companies use unstructured interviews which are practically useless in identifying good candidates. The solution is to switch to a more structured process:
• Break down the hiring decision into component parts (e.g. leadership, cultural fit, or role-specific knowledge) and focus only on data that measure those components.
• Give each rater the tools to independently assess and rate the candidates in each of the pre-determined components.
• Then, get a hiring committee to evaluate all the ratings along with additional data (e.g. test results, references) to make the final decision.
How Much Noise Should We Tolerate?
It’s hard to enforce noise reduction strategies because most people don’t even realize that noise exists. Others are concerned that such strategies may be rigid, unfair, or dehumanizing. In essence, it’s impossible and even undesirable to try and eliminate noise. More research and discussions are needed to agree on the right level of noise reduction.
The book ends off with a detailed discussion of the 7 key objections to reducing/removing noise, and whether we should regulate noise through rules, standards, or both. Both are outlined our full summary bundle.
Getting the Most from “Noise”
Biases are widely discussed because they’re more visible, and laws have even been enacted to correct/prevent unacceptable biases. On the other hand, noise is neglected because it’s often invisible, even through it has at least as much impact as biases. To improve the effectiveness and fairness of our systems, it’s crucial for organizations to start pay more attention to the costs and impact of noise.
If you’d like to learn more about noise and how to reduce it, do check out our full book summary bundle that includes an infographic, 17-page text summary, and a 33-minute audio summary.
This book explains noise in great detail from a technical, statistical, and psychological perspective. It’s packed with research and real-life examples to help us understand noise and its impact, and comes with additional appendices on noise audits, decision observer checklists, and steps for correcting predictions. You can purchase the book here. Read our Nudge summary for more intervention strategies to nudge people to toward better, healthier decisions.
About the Authors of Noise
Noise: A Flaw in Human Judgment was written by Daniel Kahneman, Olivier Sibony, and Cass Sunstein.
Daniel Kahneman is an Israeli-American psychologist and economist, best known for his work on behavioral economics and the psychology of judgment and decision-making. He’s the Eugene Higgins Professor of Psychology at Princeton University and a Professor of Public Affairs at the Princeton School of Public and International Affairs. He has received numerous awards, including the 2002 Nobel Prize in Economic Sciences and the National Medal of Freedom in 2013.
Olivier Sibony is a French consultant, academic, and author known for his work on behavioral strategy and improving the quality of decisions. He’s a Professor of strategy and management at HEC Paris. Previously, he spent 25 years in McKinsey & Company. where he was a senior partner.
Cass R. Sunstein is an American legal scholar, the Robert Walmsley University Professor at Harvard Law School and a professor at the University of Chicago Law School for 27 years. Previously, he was the Administrator of the White House Office of Information and Regulatory Affairs in the Obama administration from 2009-2012.
Noise Quotes
“Wherever you look at human judgments, you are likely to find noise. To improve the quality of our judgments, we need to overcome noise as well as bias.”
“Wherever there is judgment, there is noise—and more of it than you think.”
“System noise is inconsistency, and inconsistency damages the credibility of the system.”
“You are not always the same person, and you are less consistent over time than you think.”
“Confidence is no guarantee of accuracy…and many confident predictions turn out to be wrong.”
“Good judgments depend on what you know, how well you think, and how you think.”
“It might be costly to remove noise—but the cost is often worth incurring.”
