«Nate Silver’s The Signal and the Noise is The Soul of a New Machine for the 21st century.«—Rachel Maddow, author of *Drift*
Nate Silver built an innovative system for predicting baseball performance, predicted the 2008 election within a hair’s breadth, and became a national sensation as a blogger—all by the time he was thirty. The New York Times now publishes FiveThirtyEight.com, where Silver is one of the nation’s most influential political forecasters.
Drawing on his own groundbreaking work, Silver examines the world of prediction, investigating how we can distinguish a true signal from a universe of noisy data. Most predictions fail, often at great cost to society, because most of us have a poor understanding of probability and uncertainty. Both experts and laypeople mistake more confident predictions for more accurate ones. But overconfidence is often the reason for failure. If our appreciation of uncertainty improves, our predictions can get better too. This is the «prediction paradox»: The more humility we have about our ability to make predictions, the more successful we can be in planning for the future.
In keeping with his own aim to seek truth from data, Silver visits the most successful forecasters in a range of areas, from hurricanes to baseball, from the poker table to the stock market, from Capitol Hill to the NBA. He explains and evaluates how these forecasters think and what bonds they share. What lies behind their success? Are they good—or just lucky? What patterns have they unraveled? And are their forecasts really right? He explores unanticipated commonalities and exposes unexpected juxtapositions. And sometimes, it is not so much how good a prediction is in an absolute sense that matters but how good it is relative to the competition. In other cases, prediction is still a very rudimentary—and dangerous—science.
Silver observes that the most accurate forecasters tend to have a superior command of probability, and they tend to be both humble and hardworking. They distinguish the predictable from the unpredictable, and they notice a thousand little details that lead them closer to the truth. Because of their appreciation of probability, they can distinguish the signal from the noise.
With everything from the health of the global economy to our ability to fight terrorism dependent on the quality of our predictions, Nate Silver’s insights are an essential read.
Amazon.com ReviewAmazon Best Books of the Month, September 2012: People love statistics. Statistics, however, do not always love them back. The Signal and the Noise, Nate Silver’s brilliant and elegant tour of the modern science-slash-art of forecasting, shows what happens when Big Data meets human nature. Baseball, weather forecasting, earthquake prediction, economics, and polling: In all of these areas, Silver finds predictions gone bad thanks to biases, vested interests, and overconfidence. But he also shows where sophisticated forecasters have gotten it right (and occasionally been ignored to boot). In today’s metrics-saturated world, Silver’s book is a timely and readable reminder that statistics are only as good as the people who wield them. —Darryl Campbell
Review«Nate Silver is the Kurt Cobain of statistics… His ambitious new book, The Signal and the Noise, is a practical handbook and a philosophical manifesto in one, following the theme of prediction through a series of case studies ranging from hurricane tracking to professional poker to counterterrorism. It will be a supremely valuable resource for anyone who wants to make good guesses about the future, or who wants to assess the guesses made by others. In other words, everyone.»—The Boston Globe
«Silver delivers an improbably breezy read on what is essentially a primer on making predictions.»—Washington Post
«Silver explores our attempts at forecasting stocks, storms, sports, and anything else not set in stone.»—Wired
«Mr. Silver’s breezy style makes even the most difficult statistical material accessible. What is more, his arguments and examples are painstakingly researched…»—Wall Street Journal
«Nate Silver’s The Signal and the Noise is The Soul of a New Machine for the 21st century (a century we thought we’d be a lot better at predicting than we actually are). Our political discourse is already better informed and more data-driven because of Nate’s influence. But here he shows us what he has always been able to see in the numbers—the heart and the ethical imperative of getting the quantitative questions right. A wonderful read—totally engrossing.»—Rachel Maddow, author of Drift «Yogi Berra was right: ‘forecasting is hard, especially about the future.’ In this important book, Nate Silver explains why the performance of experts varies from prescient to useless and why we must plan for the unexpected. Must reading for anyone who cares about what might happen next.»—Richard Thaler, co-author of Nudge «Making predictions in the era of ‘big data’ is not what you might imagine. Nate Silver’s refreshing and original book provides unpredictably illuminating insights differentiating objective and subjective realities in forecasting our future. He reminds us that the human element is still essential in predicting advances in science, technology and even politics… if we were only wise enough to learn from our mistakes.»—Governor Jon Huntsman «Here’s a prediction: after you read The Signal and the Noise, you’ll have much more insight into why some models work well—and also why many don’t. You’ll learn to pay more attention to weather forecasts for the coming week—and none at all for weather forecasts beyond that. Nate Silver takes a complex, difficult subject and makes it fun, interesting, and relevant.»—Peter Orszag, former director of the Office of Management and Budget«Projection, prediction, assumption, trepidation, anticipation, expectation, estimation… we wouldn’t have 80 words like this in the English language if it wasn’t central to our lives. We tend not to take prediction seriously because, on some level, we know that we don’t know. Silver shows us how this inevitable part of life goes awry when projected on a grand scale into the murky worlds of politics, science and economics. Dancing through chess, sports, snowstorms, global warming and the McLaughlin Group, he makes a serious and systematic effort to show us how to clean the noise off the signal.»—Bill James, author of The Bill James Baseball Abstracts
What happens in systems with noisy data and underdeveloped theory—like earthquake prediction and parts of economics and political science—is a two-step process. First, people start to mistake the noise for a signal. Second, this noise pollutes journals, blogs, and news accounts with false alarms, undermining good science and setting back our ability to understand how the system really works.
friend Anil Kashyap, who teaches a course on the financial crisis to students at the University of Chicago, has come up with a simplified example of a CDO, and I’ll use a version of this example here.
