This just in from Institutional Investor: many backtests fail in real life. They write:
Makers of indexes often fill in the blanks with historic data on the components to produce hypothetical index performance. But a recent Vanguard study found that a large percentage of these hypothetical, back-filled indexes that had outperformed the U.S. stock market didn’t keep up after they went live as the index returns subsequently fell. What may be happening, says senior Vanguard ETF strategist Joel Dickson, is that indexes are being developed by “rearview mirror investing,” that is, through selection bias of what worked well in the past. The result can mean a nasty surprise for investors.
Pretty much anyone can do data mining with the computing power available on a desktop computer. And index providers will continue to do data mining as long as investors ram money into products with lousy backtests.
Back-filled index funds attract on average twice the cash flow in the initial launch phase than funds with new indexes that don’t have such data, indicating that the availability of a track record makes the fund more attractive — even if it probably won’t last.
Good backtesting can be very useful and can give investors a good idea of what to expect in the future. But how can an investor tell if the backtest is any good or not?
One thing to examine is how robust the index methodology is. For example, when we built our Systematic Relative Strength products, we subjected them to Monte Carlo testing for robustness. That made it apparent that the systematic investment method itself was sound, even though the range of outcomes on a quarterly or annual basis can be significant.
With the proliferation of indexes for ETFs, it’s becoming important to be able to evaluate how robust the backtesting was. Probably partly because of a robust backtesting process, our Technical Leaders Index has outperformed the market since inception. I’m sure many other indexes are thoughtfully constructed—but I’m just as sure that there are some that are not.
Do your homework before you put client money at risk.
See www.powershares.com for more information. Past performance is no guarantee of future returns. A list of all holding for the previous 12 months is available upon request.