Everyone in the financial services industry has seen awesome-looking backtests for various return factors or trading methods, but most people don’t even know what survivorship bias is. When I see one of those amazing backtests and I ask how they removed the survivorship bias, the usual answer is “Huh?”
A recent post by Cesar Alvarez at Alvarez Quant Trading shows just how enormous survivorship bias can be for a trend following system. Most people with amazing backtests, when pushed, will concede there might be “some” effect from survivorship. None of them ever think it will be this large!
Here, Mr. Alvarez describes the bias and shows the results:
Pre-inclusion bias is using today’s index constituents as your trading universe and assuming these stocks were always in the index during your testing period. For example if one were testing back to 2004, GOOG did not enter the S&P500 index until early 2006 at a price of $390. But your testing could potentially trade GOOG during the huge rise from $100 to $300.
Rules
- It is the first trading day of the month
- Stock is member of the S&P500 (on trading date vs as of today)
- S&P500 closes above its 200 day moving average (with and without this rule)
- Rank stocks by their six month returns
- Buy the 10 best performing stocks at the close
Source: Alvarez Quant Trading
(click on image to enlarge to full size)
Mind-boggling, isn’t it? The fantastic system that showed 30%+ returns now shows returns of less than 8%!! (The test period, by the way, was 2004-2013.)
Unfortunately, this is the way much backtesting is done. It’s much more trouble to acquire a database that has all of the delisted securities and all of the historical index constituents. That’s expensive and time-consuming, but it’s the only way to get accurate results. (Needless to say, that’s how our testing is done. You can link to one of our white papers that additionally includes Monte Carlo testing to make the results even more robust.) By the way, the pre-inclusion bias also shows very clearly how the index providers actually manage these indexes!
Mr. Alvarez concludes:
People often write about systems they have developed using the current Nasdaq 100 or S&P 500 stocks and have tested back for 5 to 10 years. Looking at this table shows that one should completely ignore those results.
When looking at backtested results, it often pays to be skeptical and to ask some questions about survivorship bias.
[…] […]