Don't mistake attribution analysis for prediction

April 1, 2018
Yoshiki Obayashi

Allocators must take care not to mistake descriptive analysis for prediction. Just because something involves numbers and metrics does not make it “Moneyball for investors.” Common attribution-style analysis is silent on the degree to which past outcome was achieved by skill or luck, and whether it will persist in the future.

When a company reports earnings, any seasoned analyst can break down revenues and expenses by business line and geography, plot historical trends, calculate accounting metrics, and so forth. Surely some do it better than others, but there’s no right or wrong when the task at hand is to describe what happened. But if you ask the same analyst to then forecast the company’s next quarter profits, that’s when the challenge begins, and predictions can definitely be wrong. The former exercise is mechanical and deterministic, whereas the latter demands to know “so what?” 

The same chasm exists between attribution and prediction when analyzing managers. Attribution analysis breaks down exposures to factors and geographies, plots historical trends, and produces numerous metrics. It tells investors the manager had a Sharpe of 0.6, a momentum beta of 0.3, and exited positions too early by an average of 5 days. That feels vaguely informative, but what does it tell us about future performance?

Let’s establish some logical building blocks to tackle this question:

1. A skill-less manager can achieve good outcomes by luck 
2. With countless active managers in the industry, many will achieve good outcomes even if none are skilled
3. Good track records achieved by luck are not expected to persist subsequently
4. For good past outcome to be indicative of the future, it must have been achieved by skill

Putting 1, 2, and 3 together explains why return chasing ends with wasted management fees. This is just one manifestation of the dangers of failing to distinguish between descriptive and predictive analyses. 

Outcome-based analysis is silent on skill vs. luck

Attribution analysis is hopeless when it comes to distinguishing skill from luck because it is just an elaborate way of describing what happened. The trouble is that outcomes in active investment management are a product of skill and a large dose of randomness, i.e. luck, and any outcome can be achieved by infinite combinations of the two.

To illustrate this point, here is a chart showing two outcomes as measured by cumulative returns. One is of a skilled equity long-only manager and the other is the result of randomly-generated trades within the same mandate.

Can you tell the difference? Neither can any return-based analysis, which will yield the exact same Sharpe, volatility, value-at-risk, alpha, factor exposures, and so forth, for the two return series.

The same phenomenon carries over to holdings-based measures of outcomes. The manager had a hit rate of 52% on the long side, sized shorts better than equal weight by 6%, and called market direction right on 55% of the days? Hand me any old laptop and I will find you randomly-generated portfolios with the same metrics in no time. It may be tempting to think that good holdings-based metrics are proof of skill, but they are not; they are still just descriptions of outcomes based on position data instead of returns, which could have been achieved by skill or luck.

One might protest this line of reasoning by saying that an allocator is unlikely to find a manager that realized so many positive metrics by chance, but this would be misguided because of building block 2 above.

Calling attribution analysis hopeless in separating skill from luck is perhaps not fair because the truth is that it doesn't even attempt at it. Combine this with the building blocks from above, and the answer to the question of what attribution analysis can tell us about future performance is clear: not much.

Attribution analysis has its merits

All of this is not to say that allocators are wasting their time and money on outcome-based analytics. Just as attribution analysis is silent on the future, our work on prediction is silent on what outcomes the allocator should desire. At the end of the day, if the outcome does not suit the allocator’s investment objectives, whether it was achieved by skill and will persist is irrelevant. 

Attribution analysis helps allocators decompose past performance to assess its fit with their investment objectives and consistency with the manager’s claims. In doing so, however, investors must always fight the common psychological inclination–known as attribution bias–to equate good outcomes to evidence of skill. An effective safeguard is to ask yourself when seeing a good metric, "how do I know the manager didn't just get lucky?" 

The views and opinions expressed in this article are those of the author(s) and do not necessarily reflect the views of Applied Academics LLC or any of its affiliates.  

Photograph by Mikihiko Obayashi