The future of prediction

ideas

January 01, 2012|By Leon Neyfakh

Taking stock of the new year means contemplating a blank slate. It means staring out into the unknown, more acutely aware than usual that we don’t know what will happen next. As we look ahead, we brace ourselves for the fact that every day of the next year will bring news. People we know will get engaged. There will be elections, military battles, and natural disasters. The world will change and our lives will change, and when it’s over we’ll look back and wonder if we could have seen any of it coming.

The desire to know what’s next is a powerful urge. History and myth are rife with examples of people trying to predict the future — medieval astrologers reading the stars for clues, ancient Greeks coming to hear Apollo speak through the oracle at Delphi, Romans performing divination rites by interpreting sheep entrails. And today, whole industries exist that aren’t too different: Psychics advertise two-dollar-per-minute hotlines, newspapers print horoscopes, palm readers set up booths at street festivals.

Dubious as it may seem in those contexts, the dream of prediction also attracts a very different breed of prognosticators: one armed not with sheep guts, but with the tools of math and science. Instead of tea leaves or Tarot cards, they wield hard drives filled with data. They conjure up systems, not fantasies. By starting with information about what has already happened — immense quantities of it — and finding inventive ways to interpret it, experts in fields from public health to national security are building increasingly sophisticated predictive models, taking advantage of new technology and new ideas about how the world is organized to push the frontiers of what we can predict.

The idea that you can systematize forecasting isn’t entirely new, especially in the realm of business and finance — an economics-oriented industry group called the International Institute of Forecasters is hosting its 32d annual meeting in Boston next June. But recent years have seen an explosion of interest and creativity in the realm of data-driven soothsaying, and some in the field predict — well, they think — that they are on the cusp of something big.

“We are at a different place in analysis than we were before. We know how to find trends, we know how to handle data in [new] ways,” said Thomas Wallsten, a psychologist at the University of Maryland at College Park who is helping to develop a cutting-edge model for prediction that relies on crowdsourcing. He added, “We’ve become much more systematic.”