The Efficient Market Hypothesis has been used as a one-size-fits-all explanation for how financial markets work. The idea that “markets are rational” justified much of the Laissez Faire policies which allowed global markets to get terrorized in 2008. However, Justin Fox’s outstanding bookThe Myth of the Rational Market: A History of Risk, Reward, and Delusion on Wall Streetferrets out the false underlying presuppositions of our financial world view.

I had the opportunity to do a long form interview with Justin — much too long for the attention span of my beloved blog readers. So, below Justin discusses the myth of financial market efficiency. At the conclusion, please add yourself to our email list to read the full interview in my upcoming book Interviews with the Brightest Minds on Wall Street. Without further ado …

Damien Hoffman: Justin, how did you build the framework you use to write about business and economics?

Justin:  While I was at Fortune in the mid-’90s, financial markets seemed to be doing great things and the US — which had a more financial-market-focused economy — was doing better than the bank dominated economies like Germany and Japan.  I bought into the whole idea that financial markets are pretty good things and free markets work in the end.

One of the things I ended up doing at Fortune was covering the intersection between corporate management and financial markets.  There were lots of different debates and phenomena about corporate America becoming very interested in stock prices and the stock market.

I wrote an article in ’97 about the phenomenon of managing earnings: moving earnings from quarter to quarter, meeting the consensus forecasts, and spending a lot of time negotiating with analysts to make sure the consensus forecast is something companies can meet or beat.  I looked into a lot of academic research done by accounting professors.  There were lots of articles pointing out evidence that corporate managers went to great efforts to manage their earnings — to keep them smooth and meet earnings targets.

This was problematic because the assumption was markets were perfectly efficient and there was no way that tweaking earnings would have any impact on a stock price.  Supposedly, the markets would see through that behavior.  That struck me as a little strange.  No one had even investigated whether managing earnings helped a stock price.

Not long after I wrote that article, a flood of research started coming out to examine why people were managing their earnings.  The efficient market assumption was finally being challenged.  Since then I have spent a great deal of time applying this information to my framework and writing.

Damien: Which intersects with your highly acclaimed book, The Myth of the Rational Market. Can you further explain this myth?

Justin:  First, we must differentiate the financial markets from other markets — for example toothpaste, potatoes, or whatever.  One of the big mistakes financial economists made in the ‘60s and ‘70s was looking at the financial markets and concluding, “Oh wow! These markets must be much more efficient than those for goods because they are much more liquid and their prices change much more often.”  Those economists were missing that financial markets, for the most part, are prediction markets.  Markets are groups of people speculating about the future earnings of a company, the future income from a loan, or cash flows from some derivate.

When the entire market is guessing about the future, it’s very hard to define “correct” or “rational”.  Financial markets can’t make rational decisions unless you have a pretty good framework of knowledge.  It’s fairly easy to make a rational decision about what kind of toothpaste to buy because it’s something that’s repeated through your life.  If you buy a different brand and you don’t like it, you go back to the one you like.

However, in financial markets the future never exactly repeats the past.  So, it’s much harder to articulate the perfectly rational strategy.  The rational action can be different things at different times.

People are not necessarily crazy.  Individual investors can be either rational or irrational.  In hindsight we can concoct a theory of rationality.  However, even at that point , the moods of the market helped create the reality at which you’re looking.

So, it’s all very recursive and makes my head hurt — and other things, for that matter [Laughing].  So, my sense is we ought to still have free financial markets, but we need to recognize that they’re always going to be prone to bubbles and bursts — mass hysterics and mass panics.

Damien: If markets have repeatedly proven irrationality and inefficiencies arise every day,  why do think we ended up clinging to the University of Chicago school of thought like an extremist religion?

Justin:  First of all, some people are still clinging to it.  People like having very simple explanations of the world.  However, in reality, different explanations work at different times.  I don’t think there is any simple theory that explains it all.

Second, from the period of the late ‘70s through 2008, their theories seemed to be working reasonably well.  Meaning, letting financial markets set the agenda seemed to work for the US economy.  There were people from the beginning of the ‘80s who were complaining about the rise of Wall Street and how much debt Americans were taking on.  But the economy kept growing and a lot of people did really well.  So, people stuck to that world view because it seemed to be working.  That was Alan Greenspan’s explanation when he testified before Congress last November.

Damien: Now that Greenspan has testified and the cat is out of the bag with this grand experiment — that it didn’t work out as planned — who do you see stepping into the void with new theories?

Justin:  I don’t think we know yet.  I definitely see reasons for understanding that the efficient market theory explains only certain things and not a lot of other things.  However, I don’t see a new explanation that explains everything.

When I started working on the book, I thought behavioral economics and behavioral finance might answer a lot of questions using the insights of psychology and empirical research into individual investing.  But those theories haven’t offered very good explanations for why we had a financial panic last year.  So, I ended the book in a little bit of a muddle because I’m not really clear what the new paradigm will be.

There are a lot of people trying to take insights from physics and other studies of adaptive systems in biology and elsewhere.  Maybe they’re going to get a great model at some point, but they are not there yet.  So, we’re in a situation where the old theory, despite all its flaws, might stick it out for a while yet.