How do we evaluate Robert Shiller's forecast?, by Scott Sumner

Nobel Laureate Robert Shiller recently made some comments about Bitcoin:

When it comes to economic bubbles, there is perhaps no single greater authority than Yale economics professor Robert Shiller. Shiller wrote a seminal book on speculation and its devolution into mania, called Irrational Exuberance. The book analyzes bubbles throughout centuries. Shiller has won the Nobel prize for his work in the area. Among his other contributions, he has created tools and indexes for measuring price levels and valuations. So when Shiller suggests that the bitcoin space might be a bubble, investors would be wise to take note. “Irrational Exuberance” and Bitcoin In an interview with Quartz, Shiller said that the “best example right now” of irrational exuberance is bitcoin.

Is it true that investors “would be wise to take note”? More importantly, how would we evaluate that claim? I don’t think we as a society know how to evaluate these claims, and indeed I think we do so in the wrong way.

One solution is to simply wait until we see what happens. If the price falls sharply, then that would confirm Shiller’s prediction. If it rises further, or even levels off for many years and decades, then he’s wrong. But I see a couple problems with that common sense approach:

1. The returns are likely to be skewed. To take the most obvious example; it’s much more likely that bitcoin will rise from $5000 to $15000, than fall from $5000 to negative $5000. Indeed there is a zero lower bound on bitcoin prices.

Now suppose that markets are efficient and bubbles do not exist. Also consider an asset with a very uncertain value, because its fundamentals are not well understood—like a new privately issued fiat electronic currency. Now go back to when the price of bitcoin was $30, and people were already calling it a bubble. Investors presumably knew there was some probability that bitcoin would catch on in a big way and soar much higher in value. They also knew that the price could never fall below zero. So it might go up to $1000 or more, but will definitely not fall more that $30. Sounds like a great investment, right? No, because if markets are efficient then the probability of it falling back toward zero must be many times higher than the probability of it rising up to $1000 or more. We just happen to live in a universe where a very improbable event occurred; its valued soared more than 100-fold (indeed more than 1000-fold if you go back even earlier in time.)

My point is that when people predict that bitcoin type investments are bubbles, they are usually going to appear to be correct in retrospect, even if bubbles do not exist. That’s simply an implication of the skewness in the returns. A low probability of high upside, and a high probability of modest downside. So being “correct” actually doesn’t prove very much.

2. A second problem is related to “data mining”. To his credit, Shiller makes very public predictions that can be discovered using Google. Thus I quickly discovered a January 2014 article where Shiller claimed that Bitcoin was a bubble:

Shiller is convinced bitcoin is a bubble and he is bemused by the fascination surrounding digital currencies. He said he is amazed by how people are excited by bitcoin – and bear in mind that a man who won the Nobel Prize for his work in the field of behavioural probably isn’t easy to surprise, let alone amaze. Shiller said:

“It is a bubble, there is no question about it. … It’s just an amazing example of a bubble.”

It would be rather presumptuous to argue with a Nobel laureate, but luckily someone already did that. Back in December Forbes put Shiller’s work to the test, comparing his findings with the bitcoin bubble.

Forbes contributor Tim Worstall argued that preventing a bubble from forming in the bitcoin market is not easy, since the market is not developed and lacks many tools needed to detect a bubble.

However, Worstall said it is “more than likely” that we are in a bitcoin bubble.

I don’t have a big problem with Worstall’s “more than likely” remark, given the skewness discussed earlier. But I do have a problem with Shiller’s claim. At the time, bitcoin was at over $800. A year later it had fallen to $200. Good call, right? Not quite, because today it’s over $4000. How can there be “no question about it”, when three years later bitcoin at $800 looks like an absolute steal? Were investors “wise to take note” of Shiller’s 2014 call? Not if the alternative was buy and hold until today.

To summarize, Shiller made a bubble call in January 2014. Most bubble calls for assets with highly asymmetric returns will look “correct” in retrospect, even if markets are 100% efficient and bubbles do not actually exist. And that’s true of people who only make a single bubble call.

But Shiller is making repeated bubble calls, just as he talked about irrational exuberance in the stock market in both 1996 and 2000. So which prediction do we hold him accountable for? If Bitcoin falls in half over the next 12 months, does that prove his 2017 prediction correct, or does it show his 2014 call to be incorrect? (prices would still be far above 2014 levels.) How do we evaluate the record of bubble forecasters who keep making one bubble call after another, until the highly volatile asset finally has a big price drop?

Here’s my claim. Bubble forecasts for highly volatile assets are not very useful, because of the reasons discussed above. These bubble calls do not represent good investment advice, and we have no reliable way to test them, even in retrospect. We do have some methods for testing for the existence of bubbles, but they need to be far more systematic. In fairness, Shiller has also done some of those more systematic tests, and found some intriguing results (such as what looks like excess volatility in US equity indices.) Even those tests can be challenged, but they are at least suggestive.

But these recent bitcoin bubble calls are virtually untestable. I don’t believe bubbles exist, however I also think the probability that Bitcoin prices will be lower in 2020 than today is much greater than the probability that prices will be higher. But that doesn’t mean it’s a bad investment, because prices might be vastly higher. And if lower, that provides very little support for bubble theories, because that outcome is likely to occur even if bubbles don’t exist.