Can a quant model really be trusted?
For all kinds of reasons, investment managers tend to find quantitative models to be interesting, but insufficient. They want to have access to the outputs of quantitative rankings, but then to only use that output as one consideration among many when making the ultimate decision about what to buy or sell. Not surprisingly, human emotion ends up getting the biggest say.
Maybe, there are some problems with that approach. Consider the following study (Dresdner Kleinwort Macro Research):
The first study I want to discuss is a classic in the field. It centers on the diagnosis of whether someone is neurotic or psychotic. A patient suffering psychosis has lost touch with the external world; whereas someone suffering neurosis is in touch with the external world but suffering from internal emotional distress, which may be immobilizing. The treatments for the two conditions are very different, so the diagnosis is not one to be taken lightly.
The standard test to distinguish the two is the Minnesota Multiphasic Personality Inventory (MMPI). This consists of around 600 statements with which the patient must express either agreement or disagreement. The statements range from “At times I think I am no good at all” to “I like mechanics magazines”. Fairly obviously, those feeling depressed are much more likely to agree with the first statement than those in an upbeat mood. More bizarrely, those suffering paranoia are more likely to enjoy mechanics magazines than the rest of us!
In 1968, Lewis Goldberg obtained access to more than 1000 patients’ MMPI test responses and final diagnoses as neurotic or psychotic. he developed a simple statistical formula, based on 10 MMPI scores, to predict the final diagnosis. His model was roughly 70% accurate when applied out of sample.
Goldberg then gave MMPI scores to experienced and inexperienced clinical psychologists and asked them to diagnose the patient. As the chart below shows, the simple quant rule significantly outperformed even the best of the psychologists.
Even when the results of the rules’ predictions were made available to the psychologists, they still underperformed the model. This is a very important point: much as we all like to think we can add something to the quant model output, the truth is that very often quant models represent a ceiling in performance (from which we detract) rather than a floor (to which we can add).
The Dresdner Kleinworth research finds the same conclusions in the realms of baseball, wine, university admissions, and criminal recidivism. Could it also apply to investment managers?
That is a pretty hard pill to swallow for investment managers, who as a group, don’t lack for self-confidence. Yet, for those investment managers who truly understand the quantitative model, who built the quantitative model, who have stress-tested the model, and who are using an adaptive factor (like relative strength), strict reliance upon the model may very well represent “the ceiling in performance.”
