Along the panoply of much-hyped hedge fund terms, “machine learning” stands out. The predominantly PhD-driven sales pitch makes stunning claims about a computer’s ability to “learn,” assign value to information, connect dots and adjust strategy sounds tempting. But new voices point to the reality, which is very different from the much-hyped promise.
“Machine learning” is a topic few question
When a pension fund, foundation or family office has meetings for quantitative “black-box” strategies, there is a troubling commonality noted. The meeting with a PhD-led team with lettered credentials following their title so long that an allocator not dare question their brilliance is what perhaps most poignantly underscores the strategy description.
Seldom are investors asked to use critical thinking. Amy Elefante Bedi, Director of Hedged Strategies at Washington University Investment Management Company in St. Louis, is different. She publicly complains about hour long meetings where the fund manager talks but nothing is said.
In large part, “machine learning” fits into this investment category. Institutional allocators are expected not to question the PhD elite, for their minds operate at a more sophisticated mathematical level than mere mortals. It isn’t until using core performance driver analysis techniques that the truth is revealed and those not wearing bathing suits in the ocean are exposed at low tide.
Machine learning systems do reveal positive insights into markets, but the little-discussed secret is that it is the human mind that is required to put the strategy together.
Machine learning is based on the concept the computer independently identifies, assimilates and assigns value to information
“Machine learning” and artificial intelligence that independently assesses information, assigns it a value and then makes trading / investing decisions is all the buzz. A recent Wired Magazine article touted Numerai, an SEC-registered hedge fund “that makes trades using machine learning models.” They are joined by the likes of Sentient Technologies, an investment firm that boasts to Bloomberg that it works by filtering “billions of pieces of data, spot trends, adapt as it learns and make money trading stocks.”
Focus on the last part of that sentence. The computer automatically “adapts as it learns” while “making money trading stocks.”
Such is the almost manic buzz and along with investor interest. But is the promise of “machine learning” more fad than fact?
While there is not a universally approved definition for machine “learning” in a hedge fund set, according to two quantitative development experts the definition looks something like this:
Machine learning concept is about a computer examining information – analyzing often vast amounts of data – and then the computer assigns allegedly meaning, relevance to that information without human assistance or background knowledge. Then the machine learning system makes investment decisions. It is this end to end “investment robot,” if you will.
Joey Krug, a Thiel Scholar who is developing prediction markets and currently sits on the board of advisors at Numerai, as well as Quantopian’s Dan Dunn, both agreed to this general definition.
Machine learning falls down based on three precepts
There are three premises upon which to question machine “learning.”
First, the “learning” concept exceeds the core limitations of if-then Boolean logic, upon which all computer systems and mathematical formulas are based.
Second, for an algorithmic system to be durable, it must be driven by economic or market rationale, which is noticeably absent from the quant-led discussion.
Third, while the PhD ability to speak for nearly one hour in a presentation and not say anything of substance is impressive, the actual audited performance has been anything but spectacular. This could be because machine learning systems violate the core rules of algorithmic trading / investing formula development.
If-then Boolean logic is not as expansive as the human mind
It is difficult to discern where, exactly, algorithmic investment methods developed from. It is documented, however, that the managed futures CTA hedge fund segment was first using and touting the benefits of a systematic, mathematical approach to the markets back in the 1980s.
After the turn of the century, with electronic markets picking up steam, it is documented that derivatives exchanges such as the CMEGroup pioneered the and advanced the concept of electronic market places and there was a loosely created school of thought on how markets should operate and derivatives regulated.
What isn’t documented or taught in PhD quantitative classes is the methods upon which algorithmic trading strategies were developed because it was considered proprietary knowledge. This school of thought, one of many, outlines core principles of how an algorithm should be developed and many of today’s Silicon Valley algorithm developers are ignoring these core precepts – and it shows in their audited performance.
Humans learn through if-then as well as non-linear logic
The first rule of algorithmic development is to understand the math-based limits of algorithmic thinking. One hole in the machine learning concept is that the limits of mathematical logic make it difficult for a computer to assess non-linear information. Humans often learn – and develop investment thesis – by connecting non-linear dots. A computer’s rigid method of “learning” isn’t really learning and can only lead to limited knowledge assessment, a critique which Krug says “is not invalid criticism.”
All algorithms are based on math. All math features definitive Boolean if-then statements. If 2 is subtracted from 5 it results in 3, for example. All formulas are if-then statements.
When considering investing, however, there are a significant number of non-linear thinking that goes into discovering insight. Two examples were provided: Brexit is an example where algorithmic analysis of the news media and most research would have led the investor to the wrong conclusion on the vote outcome and more importantly the resulting market reaction. Another example is the analysis that took place in 2015 before the August stock market crash. Balyasny Asset Management was in touch with ValueWalk leading up to the crash, discussing strategy probability paths. They made the determination that markets would be negatively impacted by the initial withdrawal of center al bank stimulus, which was not much discussed in the general media or public domain at the time. They were correct and avoided losses in the algorithmically predicted August flash crash. In both cases, an algorithm could not “learn” the real meaning of information based on public data much less assign value and then adjust trading strategy.
There is too much emphasis on “learning” in machine learning
Understanding how investment thesis is developed, Dunn says there is too much emphasis on the word “learn” in “machine learning.”
The first part is about the useful word “learn.” It feels like you are having a semantic argument with what “learning” actually means. In the machine learning sense, they are not talking about “learning” in the sense it is something that has passed the Turing test (a measure of a computer’s ability to mimic humankind) regarding gaining knowledge. They are talking about building up a base of past behavior knowledge for future predictions. The word “learn” may be overloaded in the term “machine learning.”
Most people would not disagree with you about your definition of “learn.” When algorithmic developers use the word “learn” in “machine learning” it “means something less robust,” according to Dunn.
For an algorithm to be sustainable, it must have a fundamental economic or market justification or it is luck to a large degree
The word “learn” has a different definition when used in the context of “machine learning” than it does in most human settings. While a significant degree of if-then decision processes are involved in human learning, it is the often difficult to connect often obfuscated economic dots in an investment thesis that differentiates an approach.
The second core algorithmic rule development violation machine learning hedge funds seem to perpetrate is that their formulas are seldom, if ever, explained based on an economic rationale or markets-driven supply and demand logic.
“Most people using machine learning are not going to necessarily have an economic logic or they don’t explain it from this standpoint,” Krug said. “Many quants do not understand the economic rationale.”
The problem can be located in two types of formula development strategies: strategies driven by nothing but past performance statistics and market trends that occur as a result of a repeatable economic rationale or a somewhat consistent market-based truism. If a formula is not backed by a fundamental underpinning, it can be considered luck-based to various degrees.
Dunn notes that finding the economic or market understanding for why an algorithm works “is a very tricky issue.” Quantopian, for instance, looks to see if the quantitative logic can be supported by economic or market rationale before investing, calling it “a more challenging requirement.”
Audited performance speaks louder than a PhD degree and a superior attitude
The third reason to question machine learning hedge funds is the lack of audited, public success.
One can assume that when core precepts of algorithmic formula development are broken, performance becomes more challenging to attain.
With machine learning, while there have been proclamations made in the media that the systems “learn” and are “making money trading stocks,” there has yet to be any audited, public performance to substantiate that claim.
While Dunn “cannot point to specific audited performance” to support machine learning hedge funds, performance of one of the rare funds to publish performance is the CTA KLA Capital. According to CTA Intelligence, the fund lost 28% in 2016 after dropping 12.4% in 2015. (CTA hedge fund performance is required to be reported on a consistent basis and is subject to performance audits by the National Futures Association.)
While Sentient Technologies makes the claim that ““The AI system evolves autonomously as it gains more experiences…” the firm does not disclose audited performance to the public. The same is true of another hedge fund proclaiming the virtues of machine learning, Numerai.
Machine learning is adding value to investment research, but the “machine” is just not doing much “learning”
While the promise of machine learning at present does not meet up with the reality, it is critical to note that the work being done in the area is yielding positive results. The little-discussed fact, however, is that it takes a human mind to connect the dots and do the real “learning,” value assignment and interpretation of data to then create an if-then statement.
Dunn notes that many quantitative investment insights are being developed through machine learning techniques. He says the common perception that a machine can “learn,” assign a value to information and then connect dots to make a trading decision might be a little far-fetched… at this point, anyway.
Two Sigma, a leading quantitative hedge fund, does not publicly discuss its performance. But the registered investment firm’s published thoughts on the topic indicates value in machine learning research techniques is being found. Machine learning has value.
Machine learning has value. The idea that a computer is an investing genius without humans behind the project for non-linear interpretation, however, might be a little more than far-fetched.