This project investigated the effect of news articles on financial market trends, as compiled in lists such as the Standard and Poor’s 500 (S&P 500) index. This project was based on the premise that news articles contain specific characteristics, such as sentiment and events, that could be built upon to predict the direction of the stock market. The ultimate goal was to develop a machine learning (ML) model that could use news article features, alongside other stock market data, to forecast market trends effectively.
‘Market trend’ is defined as the tendency of financial markets to move in a particular direction, over a given time frame. The direction would be determined by a number of variables, such as government policy, events, speculation, and supply and demand. It is difficult to reliably predict these variables. However, in reality the market is made up of many investors and traders who are willing to buy or sell shares of a specific stock. The tendency, in this respect, is that investors and traders use news sources to try and predict where the stock price will be in the future and place their order accordingly, following their evaluation of the said sources. The premise, here, is that certain events are expected to lead to a positive outlook – and hence attracting more buyers – while other events lead to a negative outlook, triggering an increase in selling. This change in supply and demand would result in a change in the trend of the value of the markets.
For individual investors, reading and analysing daily news articles would be time-consuming and tedious. However, this task could become easier through artificial intelligence (AI) systems, which can observe patterns over historical data and generalise efficiently. In this project, different ML models were constructed to analyse the movement of the stock market in relation to daily news articles. These business-related articles were sourced from the application programming interface (API) of The New York Times.
Textual features from news reports were extracted and weighted using different natural language processing (NLP) methods, such as, TF-IDF and GloVe. In essence, these methods were expected result in an accurate and computer-readable representation of the textual features. The ML models were trained on these features and stock market data to predict the direction (i.e., upwards or downwards) of a stock market equity for a given time window, for example, from market open till market close.
The possibility of predicting the stock market prices has been a subject of debate among researchers, with some stating that such a task would not be feasible, in view of the market’s volatility and unpredictability. However, the rise of AI solutions has led many to challenge this claim, and the outcome of this study supports the notion of using news articles to forecast the direction of the stock market over a long period. Moreover, the results offer valuable insight into the factors that influence the stock market. In essence, this research highlights the relevance of AI in predicting stock market trends.
Figure 1. High-level overview of the AI prediction system
Student: Andrew Darmanin
Supervisor : Dr Joel Azzopardi