The first useful tool we built out of Webmind components was the Market Predictor, which reads the news and uses what it sees there to help predict financial markets. This uses only a couple parts of Webmind, linked together in a somewhat unnatural way, but so far it seems to beat the pants off of everything everyone else is doing in the financial prediction business. Actually, it works much better than I thought it would Ė proof that sometimes you do get lucky.
Investing is a lot more popular these days than it was two and a half years ago when I designed the Market Predictor, let alone back when I was a kid, when it was something hardly anyone every talked about. Today everyone trades stocks, and nearly everybody believes they can beat the market somehow -- maybe by surfing the net to find a valuable piece of research everybody else missed. Even if they canít beat the market at least they can gamble on it. When I lived in Vegas I got really baffled at the reluctance of most US states to legalize gambling Ė I figured it was because they didnít want to cut off the revenue they made from lotteries. But now people have a new outlet for the innate urge to gamble, and the government canít outlaw it!
Some people believe itís really nothing but gambling. Thereís a classic book by Burton Malkiel's called A Random Walk Down Wall Street. Malkiel argues that a blindfolded monkey, picking stocks at random, could do as well as the best professional financial analysts. Since monkeys can't read, presumably the blindfold isnít much of a handicap ... the point is that, in his view, the financial markets are not predictable.
Of course, if everyone else in the market was a blindfolded monkey, then, you could make a lot of money by entering the market and trading more intelligently. But as it is now, there are so many people trying to trade intelligently, that any insight you might have has likely been roughly simultaneously exploited by a dozen or a thousand others, and hence "priced into the market." For instance, if you have good reason to believe that IBM stock is overvalued, and its price is going to drop soon, then so do others, and these others will start selling IBM at the same time as you. But this rush to sell IBM will make the price drop. So you won't gain anything by selling it, because it won't be overvalued anymore. This is called the "efficient market hypothesis" -- the hypothesis that there is no one on earth who can systematically do better on the markets than a blindfolded monkey.
Until I began studying finance with a view toward Webmind applications, this wasn't my view of the markets at all. Rather, coming from a democratic socialist background, my perspective was that the markets were a way for the rich and well-connected to profit at the expense of the working and middle class. The markets, I was certain, were crooked. Efficient, yes -- efficient at dumping money into the laps of the richest and most dishonest individuals.
And there is a certain amount of truth to this cynical view. Insider trading is still fairly common. Maybe you have a friend who works at IBM and so you know that their new products don't work. Then you know their stock is going to drop and nobody else does. That's a way to beat the blindfolded monkey for sure. (Poor innocent beast!)
Another way, if you're an investment bank, is to create a complex financial instruments in order to get around government regulations regarding the kinds of investments mutual funds, pension funds, etc., can taken on. Risky investments in foreign currencies are cleverly disguised as investments in US government bond derivatives, and so forth, so that mutual and pension funds can "legally" buy them. Deception is everywhere. And outright illegal activities are also rampant. Mike Lissack, a friend of mine for several years, has achieved notoriety as a whistleblower. Formerly an investment banker, he turned several large Wall Street firms into the Federal Government for defrauding the government out of billions of tax dollars. He had to hide out in an FBI safe house for a while to avoid possible assassination by his former employer.
And yet, in spite of all the crookedness, Wall Street also contains a great number of very smart people, honestly trying to beat the markets by one strategy or another. The motivation is mainly greed, to be sure -- this is not an altruist's game. But status is just as much a motivator, as well as pure curiosity, and the challenge of the task. After all, even if most trends and patterns are priced into the market, what if there's one that nobody sees but you? Maybe you can recognize from public data alone that the price of IBM stock is going to drop, while no one else is clever enough to see the truth. The efficient market view is basically that no one is this much smarter or more knowledgeable than everyone else, so that the rule holds 99.999% of the time -- the markets can't be predicted. But no trader believes this. They believe that it holds for almost everyone else, but not for the best of the best. They're smart enough, knowledgeable enough -- they can see the patterns that aren't priced in.
In the efficient markets view, which is conventional wisdom among finance professors in academia, it is not worth the cost to invest in financial advice in the hope of "timing the market" by buying stocks when they are low and selling when they are high. Malkiel's book, which came out in the early 70's and pushed the efficient market perspective hard, actually had a big impact on the market. It helped stimulate the development of index funds, which simply buy a wide selection of the most reputable stocks - those that are included in standard indexes such as Standard and Poor's or Dow-Jones. Index funds have done very well for a long time, although in the last year their performance has faltered relative to stocks of smaller, hi-tech firms.
The people out there trying to beat the blindfolded monkey fall into several different categories. First of all there are "fundamental" analysts, who study the economic fundamentals that determine an investment's true worth. How much profit has it generated in the past, and how much is it likely to generate in the future? How well is the company managed? What are the future prospects for the industries the company is competing in? These are, of course, difficult things to analyze, but many analysts are well trained and do a good job. This is what Lisa, my friend and Intelligenesis co-founder, did for a living for many years, before she quit Wall Street. She was a currency analyst. Lisa certainly believed that she was recognizing genuine patterns in market behavior, coming up with things that others didn't see. But she traded herself for years, and never made it big.
Why were Lisa and her colleagues, in spite of their insights, unable to beat the blindfolded monkey? In the efficient markets view, the answer is that, the market has already taken their analyses into account, before anyone can act on the analyses to exploit them in a significant way. Thousands of buyers and sellers, acting on this fundamental information, generally arrive at a price that reflects each investment's fundamental value. Or ... maybe they just weren't smart enough!
Of course, rational analysis of companies' prospects is not the whole story. Fundamental analysis clearly is not the sole driver of the markets! Consider, for example, market crashes such as those in 1929 and 1987. Suddenly, stock prices go down by as much as a third. Certainly, the true value of the companies' assets has not fallen that much overnight. The explanation is that markets are also a psychological phenomenon. They depend, not just on objective indicators, but on how people appraise those indicators. So you get periods of enthusiasm, when people build "castles in the air," persuading themselves that the future is unlimited for certain industries. And then the bubbles burst Ė just like the moving Internet bubble I talked about earlier.
Some financial analysts believe they can beat the market by analyzing the financial trends. "Technical" analysts try to do this by charting trends in stock prices. They make graphs of trends in stock prices, and believe that they can predict turning points by studying patterns in the graphs. Some use more sophisticated mathematical models rather than simple charts, especially now that computers are available to do the computations. None of these technical wizards has, however, established a really reliable track record. According to the efficient-markets view, this is because future trends in financial prices simply are not correlated with short-term fluctuations in future prices. One can predict long-term trends, within broad limits, but this is basically what the fundamental analysts do, and the results of their analyses are already incorporated in today's stock prices.
In addition to the "fundamental" and "technical" analysts, there are "behavioral" analysts who do their best to follow trends in investor opinion. They, in effect, try to psych out the market, anticipating when the climate of opinion is about to change. This is a very subtle field, depending on hunches and gut feelings, and it is difficult to test statistically. Some of these analysts have large followings, and make a lot of money selling newsletters to people who believe in their theories. They have stories to tell of great successes in predicting major turning points in markets. But many of the most successful have gone on to make dramatic bloopers. Just by luck, a certain number of soothsayers are always going to be right, but relying on the ones who were right in the past doesn't improve one's chances in the future very much, if at all.
In the 1970's, a new group of financial analysts emerged, called quantitative analysis. Unlike technical analysis which usually involves recognition of fairly simple patterns ("after three consecutive peaks, expect a big fall" and such), quantitative analysis uses highly sophisticated mathematics to analyze the markets. Practitioners are called "quants", or, more colorfully, "rocket scientists." Rocket science is big business on Wall Street, and has led to some huge successes and huge disasters. Last year, 1998, Long Term Capital Management, a hedge fund run by some Nobel Prize-winning rocket scientists, went under and lost billions of dollars. They were trading in a way that was mathematically guaranteed to succeed -- but it didn't. They lost anyway. The real world did not agree with the assumptions of their theorems. They held a number of investments that they believed to be uncorrelated, but actually, when the Russian economy crashed and a few other bad events occurred at the same time, all of their holdings simultaneously tanked.
As you can see, financial analysis is a very difficult and competitive field, and lots of clever schemes have failed. The market mechanism itself seems to guarantee that prices stick fairly close to their true value. But, Lisa was convinced, based on her work as a fundamental analyst, that there were patterns in the daily news that a computer could exploit for market prediction. She had been pretty good at picking up market-relevant news patterns, but she felt a computer could do even better. It could read more news than her, and study it more objectively. She convinced me, back in 1997 when Webmind was just a rough design sketch and a bunch of equations and concepts, that this was a good initial Webmind application. This would be our first Webmind "killer app" -- Webmind reading the news and predicting the markets. It was wild, it was crazy, but it made sense. Market prediction is a field where a small increase in intelligence can reap tremendous rewards.
Two years later, in our preliminary studies, using historical data, Webmind has done remarkably well. I don't want to lose credibility by promising too much -- there are enough snake oil salesmen in the world, especially the world of finance. What I can say is that we have an exciting new approach to financial analysis that exploits capabilities that simply did not exist in the past. And we're now working with a major international bank to integrate Webmind's financial analyses into their own trading systems. We hope to begin making money right away; but, it will take time, a couple of years, to be able to assess Webmind's performance with any scientific validity.
If Webmind is successful in beating the markets, what this will show, I believe, is that the financial markets are not truly efficient. What they are is almost efficient with respect to human intelligence. There are enough smart humans trading the markets, that almost any inefficiency detectable by a smart human will be immediately detected, and priced in. But, there are inefficiencies in the market that are not easily detected by the human mind, but are nonetheless real. Webmind is a non-human mind, and it can detect different patterns in the market, thus exploiting inefficiencies that humans cannot. Specifically, by reading the news and using concepts it extracts from the news to predict the markets, Webmind is detecting trends in human mass psychology that humans are not detecting. This is a fascinating accomplishment in itself, even if you have no interest in using it to make money.
There are dozens of AI products aimed at financial prediction -- using neural nets, genetic algorithms, and expert-system-type rules -- but most of these offer only small performance gains over the blindfolded monkey. This because they are really not all that intelligent. As artificial intelligence technology develops, we will see more and more situations like the one we currently have with Webmind -- exploitation of patterns in the market that no one has detected before, because there never before existed a mind with the proper orientation. Of course, if everyone started using Webmind to predict the markets, then Webmind's intuitions would become priced in, and the inefficiency would be gone. You'd need a new version of Webmind, or a different kind of AI, to gain an advantage. In the financial markets of the future, the spoils may to he or she who can develop a better, more distinctive, artificial brain.
How Does WebmindDo It?
The thing thatís unique about the Webmind-based Market Predictor, as opposed to other financial AI products, is its ability to analyze and synthesize both quantitative and qualitative data. It reads both text and numbers. In addition to following statistical trends and synthesizing the results of nonlinear predictive algorithms and standard financial indicators, Webmind reads the news, just as human analysts do. We simply feed in text from readily available financial news services. The system reads this news, not in an undirected, musing kind of way, but with a particular financial data set in mind, say the Dow Jones. It constructs concepts that capture themes in the news which are correlated with what the market is going to do the next day (or the next hour, or 2 weeks later, or whatever). The financial meaning of the text is thus boiled down to a collection of numbers - one for each concept extracted, representing the relevance of that concept to the text on a certain day. The numbers corresponding to the concepts can be computed anew every day, or even more often, and used for financial analysis purposes just like numbers coming from any other source. These numbers, representing the relevancies of text-derived concepts to the news at a particular time, are what we call text indicators.
The extraction of text indicators is the crux of Webmind's financial intelligence. It relies on Webmind's ability to intelligently judge relevance, which draws on all of Webmind's abilities at reasoning, language understanding, conceptualization, and so forth. But text indicator extraction is not the end of the story. In addition to the extraction of financially relevant concepts from news, there is an additional process of learning optimal trading models for particular financial markets. Just knowing the news concepts that tend to correlate with a certain financial market (say, knowing that trouble in foreign countries tends to drive the Dow) doesn't tell you enough to make accurate predictions. You have to get at the nonlinear interrelations between news concepts and numerical patterns in the data. This has to be done differently for the Dow, for IBM stock, for the Yen, for 30 year bonds, and so forth. For each market, Webmind derives a trading model that embodies the best way of incorporating text based information into decisions about that particular market.
Webmind's trading models are what computer scientists call "Boolean automata," simple logical decision rules, just as Webmind uses for making any kind of decision. They're basically the same kind of rules that are used, within the natural language system, to decide which sense of "Java," is intended in a sentence (the computer program, the island in Indonesia, or a copy of coffee). More generally, they are the rules that Webmind follows for learning abstract concepts and categories.
For reasons of efficiency, weíve adopted a special and simplified format for trading decision rules; a format derived from the prior work Jeff Pressing, an Intelligenesis co-founder. For the last few years Jeff has been trading the Australian bond market for a group of Australian investors, using trading rules of his own invention and making a fair amount of money. Jeff's rule format doesn't make Webmind perform any better than it would if it used its default decision rule module; but it does make it learn faster. In the current configuration, the system would take about an hour to learn a trading model using the generic decision module, as compared to about one minute using Jeff's streamlined framework. This is the kind of tradeoff that you face all the time doing AI engineering: the more specialized you get, the better your performance in one particular domain, but the less generalizable the performance is to other domains. In this case, as in others, the flexibility of the Webmind software system is key: We can use specialized or general-purpose methods as the particular application demands.
Initially, when Jeff and I developed this, Lisa didn't see why this decision-rule-inferring phase of the process should be necessary. "Why," she asked, "isn't it enough just to determine the concepts, occurring in news, that drive the markets?"
My answer was simple: "People may be sheep, but they're not retarded sheep."
The serious answer, of course, is self-organization, nonlinearity. The markets are a mind, which means that even when you know what motivates them, there is a great deal of subtlety to exactly how this motivation occurs. Just because the Dow tends to be driven by trouble in foreign countries, this doesn't mean that every time there's trouble in foreign countries, the Dow's going to jump. The conditions have to be right for this connection to manifest itself. An analogy is, suppose you've figured out that a given person is a sucker for beautiful women. This doesn't mean that every time the guy sees a beautiful woman, he's going to react in a certain way. If you want to be sure of his reaction, the conditions have to be right. You want to catch the guy at a good time of day; you want to catch him when it's been at least a few hours since he was with the last beautiful woman, etc. People, and markets, are predictable, but not linearly so. They react to complex combinations of stimuli, spread out over space and time. It takes an intelligent system, like a human or a Webmind, to predict what they're going to do with reasonable effectiveness.
Table 1 shows a small selection of the hundreds of amazing results we've obtained from testing Webmind on financial analysis problems. In these simple experiments, we asked Webmind to learn optimal trading models for five major markets, first with and then without news-derived information. When it included the information derived from news archives, the performance increased tremendously, often by a factor of 3 or more. [ This data deleted ]
Under the Hood
The precise mechanisms underlying Webmind's text-based market prediction are, obviously, proprietary. They're also patent-pending: Lisa and I applied for a patent for the details of this process in mid-1998. But the basic character of the process is not a secret, as it's nothing but Webmind intelligence, applied to one particular domain.
Recall the basics of Webmind architecture. Webmind, internally, consists of a collection of software objects called "nodes," each of which contains links to other nodes, representing inter-node relationships. Some nodes contain raw data such as text or numerical time series; others are more abstract and consist entirely of links to other nodes. The design is superficially similar to a neural network, but is fundamentally different in several respects. A Webmind node is more like a "neuronal module" in the brain than it is like a single neuron. In Webmind , unlike in a neural network, link construction is carried out by a variety of intelligent software actors, and nodes and links are frequently created and destroyed as part of the learning process. Webmind 's internal intelligent actors use a variety of techniques such as genetic algorithms and statistical language processing.
Different types of nodes are used for representing different types of data. The node types most directly relevant to simple financial applications are: