Are you as sure as you think?

Why the ranges come out too narrow

You were allowed to make every range as wide as you liked. A range from 1000 to 3000 for a year is never wrong. Almost nobody does that. A wide range feels like admitting you do not know, so the range shrinks to the size of what you feel you know, and the feeling of knowing is far stronger than the knowing.

Marc Alpert and Howard Raiffa ran this game on business school students and asked for ranges they were 98% sure of. Only 57% of those ranges caught the answer. Telling people about the trap and asking again made the ranges wider, but still nowhere near wide enough.

The same game in a trading room

Every quarter for ten years, Duke University asked the finance chiefs of large American companies where the S&P 500 would be a year later, with a range they were 80% sure of. These are people paid to forecast. Over 13,300 forecasts, only 36% of their ranges caught what the market did. Their ranges were about 14 points wide. To really be 80% sure, they needed ranges about three times wider.

A fund manager who says "we expect 8% to 12% next year" is playing this game. So is every risk model that says a loss this big comes once in a hundred years. The question is never how confident the number sounds. It is how often ranges like it have caught the truth before.

Sources

1. Alpert and Raiffa, "A progress report on the training of probability assessors", in Kahneman, Slovic and Tversky (eds.), Judgment under Uncertainty (Cambridge, 1982), 294–305.
2. Ben-David, Graham and Harvey, "Managerial Miscalibration", Quarterly Journal of Economics 128 (2013), 1547–1584.
3. The chance figures assume someone who really is 90% sure on each question, with the ten questions independent of each other.