Survivorship Bias

Everyone in the financial services industry has seen awesome-looking backtests for various return factors or trading methods, but most people don’t even know what survivorship bias is.  When I see one of those amazing backtests and I ask how they removed the survivorship bias, the usual answer is “Huh?”

A recent post by Cesar Alvarez at Alvarez Quant Trading shows just how enormous survivorship bias can be for a trend following system.  Most people with amazing backtests, when pushed, will concede there might be “some” effect from survivorship.  None of them ever think it will be this large!

Here, Mr. Alvarez describes the bias and shows the results:

Pre-inclusion bias is using today’s index constituents as your trading universe and assuming these stocks were always in the index during your testing period. For example if one were testing back to 2004, GOOG did not enter the S&P500 index until early 2006 at a price of $390. But your testing could potentially trade GOOG during the huge rise from $100 to $300.

Rules

  • It is the first trading day of the month
  • Stock is member of the S&P500 (on trading date vs as of today)
  • S&P500 closes above its 200 day moving average (with and without this rule)
  • Rank stocks by their six month returns
  • Buy the 10 best performing stocks at the close

Source: Alvarez Quant Trading

(click on image to enlarge to full size)

Mind-boggling, isn’t it?  The fantastic system that showed 30%+ returns now shows returns of less than 8%!!  (The test period, by the way, was 2004-2013.)

Unfortunately, this is the way much backtesting is done.  It’s much more trouble to acquire a database that has all of the delisted securities and all of the historical index constituents.  That’s expensive and time-consuming, but it’s the only way to get accurate results.  (Needless to say, that’s how our testing is done.  You can link to one of our white papers that additionally includes Monte Carlo testing to make the results even more robust.)  By the way, the pre-inclusion bias also shows very clearly how the index providers actually manage these indexes!

Mr. Alvarez concludes:

People often write about systems they have developed using the current Nasdaq 100 or S&P 500 stocks and have tested back for 5 to 10 years. Looking at this table shows that one should completely ignore those results.

When looking at backtested results, it often pays to be skeptical and to ask some questions about survivorship bias.

5 Responses to Survivorship Bias

  1. Richelle says:

    Please let me know if you’re looking for a article writer for your site.

    You have some really good articles and I believe I would be a good asset.

    If you ever want to take some of the load off, I’d love to write some material for your blog in exchange for
    a link back to mine. Please send me an e-mail if interested.
    Regards!

    Also visit my web site: affiliate.realpassiveprofits.com (Richelle)

  2. It’s really a great and helpful piece of info.
    I am happy that you just shared this useful info with us. Please
    stay us informed like this. Thanks for sharing.

    my web site … photography (digital.cameraarticlestoday.com)

  3. May I simply say what a relief to discover an individual who actually knows what they’re talking about on the internet.
    You actually understand how to bring a problem to light and
    make it important. A lot more people really need to read
    this and understand this side of the story.
    I was surprised that you are not more popular given that you most certainly have the gift.

  4. Positivity differs from optimism in that thinking positive in the face of all situations can be used as a tool to effectively manage the health of our emotions.
    Movara Fitness Resort and Spa offers unforgettable hiking,
    fun and engaging fitness and exercise classes, incredible
    food and life changing education. The personal trainer exam includes a written component, and many a times,
    depending on the training provider, may include practical component.

Leave a Reply

Your email address will not be published. Required fields are marked *