Years ago we developed a testing protocol to help us determine how robust a strategy really is. We wanted to determine how much of the strategy’s tested returns were a result of luck and how much of the return was due to the underlying factor performance. We have run all of our strategies through that process over the years, and we published some of those results back in 2010. The data was just updated through the end of last year and the updated can be found here.
When testing a model it is always difficult to determine if the results you are achieving are repeatable or not. If you are testing a high relative strength model, for example, are the results coming from one or two stocks that make the whole test look fantastic? If that is the case I would have my doubts about how that strategy would perform in real-time. But if the results are truly from an underlying factor performance (regardless of the individual securities in the portfolio) then you have something you can work with.
The way we determine if a model is lucky or not is to run multiple simulations based on a random draw of securities. In a relative strength model we might break our universe into ten different buckets. Out of the highest bucket we might draw 50 stocks at random. We hold those stocks until they are no longer classified as high relative strength securities. Once they fall below a specific rank we sell the security and buy another one at random. If we run 100 trials we get 100 different portfolios over time. What we are trying to determine is if the individual securities in the test really matter, or is just the concept of buying high relative strength securities over time what causes the outperformance.
As it turns out, what stocks go in to the portfolio aren’t as important as exploiting the factor. A disciplined approach is that consistently drives the portfolio to strength is what drives the returns over time.
(Click To Enlarge)
The table shows the results from one of the factors tested in the paper. You can see the range of outcomes each year as well as how each model did over the 16 year test period. Sometimes the models outperform, sometimes the underperform, and some years you have mixed results. But over 16 years, all of the models outperformed! All we did was pick stocks at random out of a high relative strength basket. There is nothing complicated about it. The main thing is that the process is systematic and extremely disciplined.
More details about the testing process and results can be found in the paper (click here).
Hello,
I have been a follower of your very good blog for quite a while and frankly I am a fan. Regarding look back time frames, you model testing shows that over the years longer time frame work better. You confirmed it in another post by mentioning the effect of volatility as described in a GMO paper. it appears that 6-12 months look good in general and especially during volatile environments. However according to your model testing it clearly appears in 2003 and 2009 that shorter look back periods worked better. I think it was again the same at the beginning of 2012. It’s understandable given the sharp trend reversals we saw during those three periods of time. Such periods were characterized by a significant compression of the volatility from high levels. Going forward, would you advise to decrease the look back period during let’s say 3-6 months after a new volatility compression ? Or would you advise to use several look back periods with changing weigths? I am curious to learn more about this if you have researched it. Feel free to respond directly by email. Thank you, Henry
Henry,
Thanks for reading the blog. You hit the nail on the head when mentioned the underperformance around periods of trend reversals. Shortening the RS Lookback window would help this problem. However, this is much easier said than done. Switching parameters adds another potential source of whipsaws. You’re not going to always get it right, and when we have experimented with that type of system we found you get it wrong enough that over time your results aren’t as good.
Keep in mind that around trend reversals a trend following system isn’t going to work. I think a better idea would be to buy the laggards instead of changing the RS Lookback. Again, this is easier said than done, but it will give you more bang for your buck because the laggards (somewhat by definition) outperform the leaders during a trend change. This is also a reason why splitting the portfolio into strategies (RS and mean reversion / deep value) does so will over time.
Thanks,
John