Tweet thread by the brilliant @jposhaughnessy
1/Base Rates are Boring (And REALLY Effective)
The majority of investors, as well as anyone else using traditional, intuitive forecasting methods, are overwhelmed by their human nature. They use information unreliably, one time including a stock in a portfolio and another time
2/ excluding it, even though in each instance the information is the same. Our decision-making is systematically flawed because we prefer gut reactions and individual, colorful stories to boring base rates. Base rates are among the most illuminating statistics that exist.
3/They’re just like batting averages. For example, if a town of 100,000 people had 70,000 lawyers and 30,000 engineers, the base rate for lawyers is 70 percent. When used in the stock market, these rates tell you what to expect from certain class of stocks
4/(e.g., all stocks with high dividend yields) and what that variable shows for how that category of stocks have performed over many decades of data. We have found that since the original publication of "What Works on Wall Street" in 1996, the
5/performance of the various factors we studied has persisted. Remember that the base rates tell you nothing about how each individual member of that class will behave, rather they indicate how all stocks with high dividend yields–or whatever factor is being reviewed—will behave.
6/Most statistical prediction techniques use base rates. 75 percent of university students with grade point averages above 3.50 go on to do well in graduate school. Smokers are twice as likely to get cancer. The average 70-year old in the United states can expect,
7/based on actuarial tables, to live another 13 ½ years. Stocks with low PE ratios outperformed the market in 99 percent of all rolling ten-year periods between 1964 and 2009. The best way to predict the future is to bet with the base rate that is derived from a large sample.
8/Yet, numerous studies have found that people make full use of base rate information only when there is a lack of descriptive data. In one example, people are told that out of a sample of 100 people, 70 are lawyers and 30 are engineers.
9/When provided with no additional information and asked to guess the occupation of a randomly selected 10, people use the base rate information, saying that all 10 are lawyers, by doing so they ensure themselves of getting the most right.
10/However, when worthless yet descriptive data was added, such as “Dick is a highly motivated 30-year-old married man who is well liked by his colleagues,” people largely ignored the base rate information in favor of their “feel” for the person.
11/They’re certain that their unique insights will help them make a better forecast, even when the additional information is meaningless. We prefer descriptive data to impersonal statistics because it better represents our individual experience.
12/Then when stereotypical information is added, such as “Dick is 30 years old, married, shows no interest in politics or social issues, and likes to spend free time on his many hobbies, which include carpentry and mathematical puzzles,” people totally ignore the base rate
13/and say that Dick is an engineer, despite a 70 percent chance that he is a lawyer. This bias has been proven time and again with numerous tests over a range of subjects. People always default to making predictions based upon their individual experience and intuition.
14/It’s difficult to blame people. Base rates are boring; experience is vivid and fun. The only way anyone will pay 100 times a company’s earnings for a stock is if it has a tremendous story. Never mind that stocks with high PE ratios beat the market less than 1 percent of the
15/time over all rolling 10-year periods between 1964 and 2009—the story is so compelling, you’re happy to throw the base rates out the window. And *nothing* has changed here nor do I think it will in my lifetime. Arbitraging human behavior is the last sustainable edge.
