Forecasting and uncertainty: A survey

Abstract

The origins of forecasting can be traced back to the beginning of human civilization with attempts to predict the weather, although forecasting as a field first appeared in the 1940s and attracted more followers from the early 1950s, when the need for predictions emerged in different fields of endeavor. It expanded considerably in the 1960s and 1970s when benefits were ascertained and computers were employed to perform the tedious calculations required. But initial successes in the fields of economics and business were first moderated and later reversed, with reality checks, first during the 1973/74 energy crisis, afterwards during the prolonged economic stagflation of the late 1970s and early 1980s and further deteriorated during the severe 2007/8 global financial crisis. The initial, optimistic expectations that social sciences will (using powerful computers and sophisticated models) replicate the predictive accuracy of hard ones were repeatedly shattered. This has left diverse fields like economics, management, political and human sciences and even worse medicine with no objective evidence of successful, accurate predictions, casting doubts to their usefulness and “scientific” vigor. At the same time, weather forecasting achieved success for immediate term predictions improving its accuracy and reliability over time. This paper starts with a historical overview of non-superstition based forecasting as it is practiced in different areas and surveys their predictive accuracy, highlighting their successes, identifying their many failures and explaining the reasons involved. Consequently, it argues for a new, pragmatic approach where the emphasis must shift from forecasting to assessing uncertainty, as realistically as possible, evaluating its implications to risk and exploring ways to prepare to face it. This paper expands Rumsfeld’s classification to four quadrants (Known/Knowns, Unknown/Knowns, Known/Unknowns and Unknown/Unknowns) in order to explore the full range of predictions and associated uncertainties and be able to consider the implications and risk involved. Finally, there is a concluding section summarizing the findings, reiterating that we must accept that the great majority of our predictions will be inaccurate and that uncertainty, sometimes huge, surrounds all aspects of our future. There are also some suggestions for future research aimed at turning forecasting into an interdisciplinary field increasing its value and usefulness.