Fundamentals, Strategy

The Structural Flaw in Data-Driven Quantitative Analysis

February 11, 2013

Rodin's ThinkerPeople who lean more towards fundamental analysis tend to think differently than others using alternate methods. There is a distinction between those people that rely on news and statements by management and fundamental analysts who rely solely on numerical analysis, which are probably more like technical analysts in their thought processes. Those who take a qualitative approach tend to question their assumptions more, because they rely on opinions and obviously everybody who engages in any kind of analysis should always question the underlying assumptions and learn the limitations of their systems.

I want to make it clear that I am not deriding technical analysis as a foolhardy approach. Even the most rigorous fundamental analysis is subject to the issues I discuss here. However, the severity and prevalence of the issues for fundamental analysis is lower. Technical analysis by its very nature exposes itself to the flaws of forecasting.

The Security Blanket of Numbers

Fundamental analysts using a numerical approach are not ultra-reliant on historical analysis to the extent that they limit exposure to assumptions regarding the future, however they share their faith in numbers with technical analysts. Rarely is the world so black and white, and often you might find yourself using a mixed approach. Something about the way we are wired makes us more comfortable trusting calculated numbers than the statements of individuals. I might not be exposed to a representative sample of people, but I see this worship of numbers far too often. It could be due to the fact that most people I know are science and engineering types. If you are surrounded by people more attuned to the philosophical theory of knowledge it might be different.

A prime example is someone telling me about the virtues of the PEG ratio and its superiority over the statements of management. I am a big fan of listening to or reading earnings conference calls. I want to hear management talk. I skim the questions at the end for interesting ones, but the vast majority of them I do not care too much about. I want to know management’s take and focus, and then I go to complete articles written by investors or analysts. The reason I do not like the questions is that they are piecemeal. I am a fan of the big picture.

This individual was telling me that I should be looking at the PEG ratio. It was not the suggestion so much as the belief in the ratio that I had a problem with. I look at the PEG ratio too, but the mantra I chant is that it relies on the assumptions about growth. Nobody knows if the growth projections are going to hold water. In 2005 the five year growth projections were rosy right through the 2008 debacle. Tell someone who loves the ratio about the inherent flaws in projections and you may end up in a fist fight.

Some human’s guess at the growth rate does not warrant that level of loyalty. Even if you take an average from many analysts you will not get an accurate picture. These people rely on the same processes and assumptions. They are forced to work with what they have, so you have no need to hold it against them. However, you also have no reason to blindly believe they know what they are talking about. So much of my online reading involves downplaying the worst case scenario. The worst case is that it drops 10%. No, the worst case is that the safest stock in the world ends up declaring bankruptcy overnight due to some sort of cataclysm.

You do not need to stay up at night panicking about the unlikeliest of events, but you should not fool yourself into believing that an unlikely event will never happen. It may not always be catastrophic, but there are quite a few plausible events that could upset your plans. Even if they are improbable they are possible, so you need to structure your investing strategy to account for this.

The Problem with History

So now that we have an understanding of the flaws inherent to numerical fundamental analysis we should move to technical analysis and its exposure to the problem of history. The problem of history as I call it is the standard disclaimer you see for most investment newsletters or systems. Past results are not a guarantee of future results, or something to that effect. Technical analysis is all about historical data. The idea is that there is so much data involved, and so many past situations that were identical that you can count on a repeat performance. This is fine in the short-term, but I see it sometime haphazardly applied and time frames get too long. For example, I see people rationalize positions using technical analysis, “It is oversold now so I shouldn’t sell for a loss, because it will rebound.” Pure technical analysis of stocks cannot be stretched out for long periods.

Those who understand the limitations of the technical approach can wield it effectively. However, there is a loyalty to charting and patterns and fancy mathematical models that creates this false cloak of security and certainty.

The one great truth of patterns is that they always hold — until they don’t. This obvious statement is of the utmost importance. Extreme, unexpected events can wipe you out. This applies regardless of the type of system you prefer. Technical or fundamental, the unexpected sucks. Proper money management can mitigate this risk substantially, but very few people use a proper money management strategy.

The reason I am focusing on technical analysis in this part of the discussion is because it does not account for massive deviations from the norm. Even though all data sets have outliers, most forms of technical analysis ignore that by aggregating a lot of historical data. In the grand scheme of things outliers do not matter, but at the level of the individual they matter far more than the grand scheme. It is not helpful to the retiring individual to know that 2008 is a natural part of a 30 year super business cycle, and after a 5 year severe shake out the next 30 years will be see a new peak made and that’s why many employees decide to have their own business instead, or purchasing a existing one, you can go online to find What You Need to Know About Buying an Existing Business that serve you better for this purpose. That the retiree is likely to be deceased in 30 years does not matter to the forces of time and history, but it really matters to the retiree that their wealth has shrunk and will only begin to recover after the average life expectancy.

That severe individual events hardly matter to the grand scheme is a fact of reality. In the context of the universe, the sun is nothing special and when it burns out it’s no big deal, but it will be a real bummer for Earth. “The universe hates you, deal with it.” (Bonus points to whomever can name the source of the quote, though there may be more than one.)

The Need to Explain the Inexplicable

In our quest to create order from chaos we worship false idols. Does statistical analysis help you? Yes. Does normalization of large data sets help? Yes. Are they a guarantee? No. When you pick and choose your data sets you can prove anything you want. You hear that buy-and-hold works for 10 years, then after 10 years it turns out not to be true and people change the time frame to 20 years to make it work.

Take gold for example. Gold has gone up for the last 30 years. That is a ton of data, and gold is headed to $5,000/oz if the trend of the last 30 years continues. Has gold gone up for the last 40 years? No. On an inflation adjusted basis gold still has not recovered from the crash in the early 80s. Gold would need to be $2,200/oz or more (ballpark) to be at the same level on an inflation adjusted basis.

So is gold in a long-term recovery after a significant correction? Does the historical data help us extrapolate the future potential of gold? My answer is “not even sorta” and I feel that is true especially for gold. There are so many factors influencing the direction, and not all of them are uniform. Investment demand ebbs and flows based on risk, inflation, and many other factors. The market does not even react uniformly to the same news. Sometimes it is seen as a bad thing for the economy and gold rises as a safe haven. Other times it is seen as a bad thing for the economy and gold falls due to a drop in demand. The difference is that the bad news will hurt the global economy, but this piece of news is focused on China and is likely to mitigate their appetite for the yellow metal.

Closing Thoughts

Extreme events are not accounted for in any model and neither is the market’s willy-nilly responses to stimuli. Even the qualitative fundamental analyst has to understand that there is a world of uncertainties out there. Relying on a system that tries to project the future based on normalized historical information is asking for pain when you encounter that rare event. Charting is notorious for its patterns. Charts have become too engrained as a tool for investing that I suspect it’s a self-fulfilling prophecy — but that is a separate issue. Serious events like those from 2008 can throw a monkey wrench into the works. That does not mean we should throw out our books, but we need to understand the limitations so we can prepare for them. Again money/risk management is the easiest way. Avoid greed and use reasonable position sizes.

Long-standing beliefs should be questioned. Nothing is for sure. The less you have at risk, the less time you need to spend rolling around in the doom and gloom. Many people have that one asset that most of their money in. This “safe” asset is out of their mind. Property was not immune to an unexpected event. Gold was not immune to an insane crash in the 1980s. If you keep your money in cash, is it really safe? Wherever you are not worried is where you can be surprised most. Think outside of stocks alone when considering your asset allocations.

If you have fat bank accounts then you are open to currency risk. That is not a dead end, but it allows you to hone in on where you should look for some protection. Extreme, unpredictable events hurt most when you are not looking. If you were a property disciple then selling it off prior to 2007 or 2006 (at least 2008) would have been the best option for you. Keep the cash, when the market has its tantrum and property values are depressed buy a lot more property and ride the wave. There were signs that property was weakening, but the trend and analyst guesses at growth allowed people to feel comfortable holding on. Property is complicated to get rid of, but other assets function the similar way.

I know many gold bugs. When gold was making new highs on a regular basis they were happy. Now that gold is oscillating and bouncing off resistance there are all kinds of rationalizations for the future of gold that lead to them holding onto their position. Instead, the difficulty gold has been having warrants an extremely light exposure. If you miss the big rise, then fine. What if it collapsed again? You may have enjoyed a spike to a new high, but could you survive comfortably for a collapse into the $500 range? If the answer is no, prudence is required. History tells us that, but we are only using history as a sign to be smart not as a guide to the future. That it did happen, means it can happen; not that it will happen.

This article was contributed by “The Archivist”, founder and editor of The Market Archive.

Enhanced by Zemanta

6 thoughts on “The Structural Flaw in Data-Driven Quantitative Analysis

  1. Thanks for this excellent article Archivist. I have to admit that I often times fall into the category of those who rely too much on numbers. As you’ve pointed out that’s probably due in part to my engineering/science background.

    You make some excellent points however, particularly with respect to the fact that what are often considered to be market “disasters” are actually quite normal in the big scheme of things (but they really CAN be disastrous for an individual with a finite investment timeframe). Your discussion emphasizing the fact that future predictions based on the past are often times questionable at best was interesting as well.

    Thanks again!

    1. I use a bit of hyperbole to get my point across. Once you are aware of the flaws you can use quantitative analysis effectively. Hopefully nobody thinks I am saying don’t do it at all. Some people use numbers as a teddy bear to hug, but its a tool and has pros and cons.

    1. It is also about realizing the basic flaws underlying to components of your analysis. So often data is treated like raw materials, and the creation of the final product (the analysis) is where all the skill and craft comes in. You give me tons of clay and I will make bricks. For something physical like that it will be obvious to people that the source of clay matters as far as quality. What if I tell you that it is from the finest ever clay ever discovered, but unbeknownst to anyone right now some of that clay deposit has a very small impurity, that causes it to explode when it goes into a clay oven? You’ll likely not survive that.

      You weight fundamental and technical equally. Great, but if you think that you’re balanced approach beats out an approach that relies on one or the other, because it is “complete.” There is a world of unknowns out there.

      Another take away, a statistician will come and tell your lifeless body that the amount of the impurity was statistically insignificant against all the clay available at that site, but that does not help you.

      That clay has been used a million times and nothing like this has happened. Again, you won’t care because you’re not around anymore. History does not determine the future.

      Test your position size against failure not against success. You analyze from both schools of thought, but you are relying on information supplied by a myriad number of people. I used the PEG ratio in the article. Here is another, revenue numbers, margins, etc. Throw in some book cooking and suddenly you have flawed fundamental analysis. If you have a 100,000 account and only invest 1000 at a time, then you’ll be fine. If on the other hand technical and fundamental indicators all point to a rocket ship and you invest 20,000, you might not be. The risk might be well worth it for you, but it won’t be for some other guy. Analyze the failure of your thesis, if you are comfortable with 20k becoming 1k for the opportunity to make 500k, it is fine. There is nothing wrong with that. Too often people only calculate the success and not complete and utter failure. I use gold as an example, because I know people that use gold as an alternative savings account. Gold is volatile though, and extremely so if you look at history. Most people do not understand the risks. 10 years of fantastic growth is enough data for them.

      We use our analysis and numbers with a loyalty that they do not deserve. Continue your analysis, but factor in chaos. Chaos usually means to incorporate complete and utter failure. It also applies to unexpected success, but negative events have more power. I should do a follow up on that. Blue chips are safe you can have large position sizes. Chaos would dictate smaller positions and more diversity, because even a blue chip can halve its value if something really bad happens. It is just unlikely, but if it happens to you it won’t matter how unlikely it is.

      On a final note, all of this rarely applies to less than a few months unless we are in an extremely volatile time. Normally years are required to expose yourself to these risks. If your luck is rotten though, you could invest in Apple at 600 on Monday and see it hit 500 on Tuesday or Wednesday.

Leave a Reply

Your email address will not be published. Required fields are marked *