Prediction, The Black Swan and 9/11

Minorities set to be US majority -White people of European descent will no longer make up a majority of the US population by the year 2042 - eight years sooner than previous estimates.
Bush resolute in face of Iraq death toll -President vows to make sure American lives not 'lost in vain'
All US adults could be overweight in 40 years
Fanny Mae unveils loss of $2.3 billion - Problems in the US housing market have pushed mortgage finance company Fannie Mae into the red.

All of these headlines reveal the fraud of prediction. Why do people predict? Why do people think they can predict? To feel better about the future and because they are arrogant about what they think they know. Today being the anniversary of one of the most unpredictable events in history and with a tip of the hat to Nassim Nicholas Taleb, author of The Black Swan: The Impact of the Highly Improbable (the first chapter can be found here), I'm gonna take a look today at the fallacy of prediction, one of the all-time greatest frauds perpetrated on humanity. If you're a stockbroker, political analyst, or in any profession that relies on predicting the future, you might want to stop reading now.

The first thing we need to understand is that what we don't know is far more relevant than what we do know. We live in a world that is less understandable, explainable and therefore predictable than we think it is, the illusion of understanding. This problem is amplified by the retrospective distortion of past events; history appears more organized in history books than reality. We tend to concoct explanations for events after the fact, making them appear more predictable than they really are. Kirkegaard told us, history runs forward but is seen backward. Finally, according to NNT, (Nassim Nicholas Talim, the author of the Black Swan from here on) we tend to "Platonify", deifying the scholars, while the experts usually don't know any more than a taxi driver on most subjects. These defects lead to people having blindspots when it comes to seeing the black swan events that could happen. A black swan, or fat tail, is a large-impact, hard-to-predict and rare event beyond the realm of normal expectations. The 9/11 commission demonstrates all three of these problems. It's mandate was to provide a full and complete accounting of the attacks of Sept. 11, 2001 and give recommendations as to how to prevent such attacks in the future. Unfortunately a black swan's very unexpectedness creates the conditions for it to occur.

While much of the world can be analyzed and predicted statistically with reasonable accuracy, this type of analysis is useless outside of this realm. NNT insists on separating the world of uncertainties into two "worlds": Mediocristan and Extremistan. Basically a distinction needs to be made between scalable and non-scalable types of randomness. Mediocristan is where classic measures of statistics are relevant, particularly the Gaussian bell-curve. This includes measurements that are in a way pulled down by "gravity", where extreme outliers are impossible, things such as people's height and weight. However, in Extremistan measurement tools such as bell-curves are dangerous, where extreme outliers are not only possible, but defining, such as in the stock market, people's wealth and body counts in wars.

A little math, Gaussian style, the vaunted bell-curve you may remember from a boring stats class you once took, it looks something like this:Now this little tool can help you measure the probability of certain events occurring, the one above shows the distribution of IQ scores. With a large enough sample one can predict with different confidence levels the probability of different occurrences. Normal distribution, ie. the bell-curve or Gaussian distribution, uses the central limit theorem to give us averages or means and standard deviation to measure variability. A beautiful thing when applied to astronomical data as Gauss first did or maybe the birth weight of chimpanzees. Most observations hover around the middle while the odds of a deviation decline exponentially faster as you move away from the average. The problem begins with the application to Extremistan, and what NNT terms the Ludic fallacy, "the misuse of games to model real-life situations". Some assume that the unexpected can be predicted by extrapolating from variations in statistics based on past observations, ie. causal determinism. To the detriment of the world Gaussian thinking has been applied to many areas of Extremistan.

The failure of the bell-curve is best demonstrated by an example in a review from all places, the Wall Street Journal:
If 100 random people gather in a room and the world's tallest man walks in, the average height doesn't change much. But if Bill Gates walks in, the average net worth rises dramatically. Height follows the bell curve in its distribution. Wealth does not: It follows an asymmetric, L-shaped pattern known as a "power law," where most values are below average and a few far above. In the realm of the power law, rare and extreme events dominate the action. Wealth is an Extremistan measure. The last century has seen the rise of economics as a science, where statistical models are used to manage risk. Thus the birth of modern portfolio theory (Merton and Scholes) and monsters such as LTCM (Long-Term Capital Management). In case you missed it, this was a hedge fund that was founded in part by two Nobel laureates, the aforementioned Merton and Scholes, which failed spectacularly in the late 90's. The fund earned outstanding returns for a few years but the methods didn't take the black swan into account and were thus exposed to events like the Russian financial crisis which almost took down the entire financial system. Instead of learning from our mistake, business schools keep pumping out MBAs with the same theory and the same risk, thus the sub-prime meltdown, Bear Sterns, Fannie Mae and Freddy Mac, but the government is there to prop them back up, so they'll probably never learn.

The point is you need the right tool for the job. Just as you wouldn't use a hammer to install a window, statistical models are dangerous tools when people blindly follow them. When people start to apply statistics to social sciences greater dangers arise. Adolphe Quetelet tried to use the Gaussian curve to construct the physically and morally average man. His contemporary Karl Marx borrowed on his ideas in Das Kapital to minimize societal deviations in terms of the distribution of wealth. War used to belong to Mediocristan, but with the advent of weapons of mass destruction you now have the possibility of wiping out huge numbers of people with the press of a button, the black swan. There's no use in predicting the next 9/11, worse yet are the methods that have been used in the 'war on terror', as instead of lowering the chances of another such event, everything done so far has acted to increase the odds.


By assuming that rare events won’t happen, banks and others almost always win; but if you lose, you can lose everything. If you've read this post all the way to this point and still don't know what I'm trying to say, go and read the book, it's a good read, and relevant to your life if you've got a bank account, insurance policy or a pension fund. Try as we might, we can't control randomness, the black swan will always exist. It could be caused by a hurricane in New Orleans, a credit crunch or the beating of a butterfly's wings on the other side of the planet, they will happen.