Improving Predictions in Deep Learning by Modelling Uncertainty

Axel Brando

d&a blog

At BBVA we have been working for some time to leverage transactional data of our clients and Deep Learning modes to offer a personalized and meaningful digital banking experience to clients. Our ability to foresee recurrent income and expenses in an account is one the most distinctives in this industry. This kind of forecasting helps customers to plan budgets, act upon a financial event, or avoid overdrafts, while reinforcing the concept of “peace of mind” a bank such as BBVA wants to herald.

The application of Machine Learning techniques to predict that an event is recurrent, the amount of money involved allowed us to develop this functionality. As a complement to this project, we at BBVA Data & Analytics are investing in research and development to study the feasibility of Deep Learning methods in forecasting, as we have explained previously in “There is no such thing as a certain prediction”. The goal wasn’t just to improve the current system but to generate new knowledge to validate this new techniques.

As a result, we have observed that Deep Learning contributes to reduce the errors in forecasting. Nonetheless, we have also have seen that situations remain in which some expenses are not predictable, and that simple Deep Learning for regression does not offer a mechanism to determine uncertainty and therefore to measure reliability.

Making good predictions is as important as detecting the cases in which those predictions have an ample range and therefore we would like to be able to include that uncertainty in the model. This would be useful to use clients not only reliable predictions, but to prioritize actions related to the results shown. That is why we are now researching Bayesian Deep Learning models that offer us a measure uncertainty related to the forecast.

Measure uncertainty to help clients

The detection of confident user behaviours in order to predict requires an analysis of the concept of uncertainty of the prediction. But what are the sources of uncertainty? Although the clarification of this concept is still an open debate, in Statistics or Economics sometimes it is classified in two categories: The aleatoric uncertainty (i.e. the uncertainty that appears due to the variability of the different possible correct solutions given the same information to predict) or epistemic uncertainty (i.e. the uncertainty related with our ignorance of what is the best model to use to solve our problem or even our ignorance respect new kind of data we did not saw in the past).

From a mathematical point of view, we could try to find a function that given a certain input data (the historical transactions of a certain user) would return the next value of the movement time series in the most accurate way possible. Nonetheless, there are limits to this approach: in our case, given the same information -past transactions-, the results does not necessarily be the same. Two clients with the same past transactional behaviour do not obviously implies same transactions in the future.

The next figure visualizes the concept of uncertainty, and try to answer the question of what would be the value of the red dot in a certain time interval.

In this case a model with or without uncertainty would predict the following outputs:

In order to approximate this function, a good state-of-the-art solution for this problem are Deep Learning algorithms due to their capacity to fit very complex functions to large datasets. The standard way to build Deep Learning algorithms provides a “pointwise” estimate of the value to predict, but does not reflect the level of certainty of the prediction. What we are looking for here is to obtain a probability distribution model of the possible predictions by using Deep Learning algorithms.

We can model two types of aleatoric uncertainty: homocedastic uncertainty, which reflects a constant variance for every client, and heterocedastic uncertainty, which accounts for the variance of each individual client given its transactional pattern.

In any case, the usefulness of the aleatoric Deep Learning, versus the epistemic Deep Learning, is greater and we detect clear improvement in our ability to have a measure of confidence of each prediction, especially because we can gauge the variance.

The type of uncertainty that we have taken into account is very useful when we have to create a new model that can forecast income and expenses. In the following plot each point is an user plotted with its respective real error prediction versus uncertainty score predicted. Further, we see that most yellowish points (where there are most part of the clients) have a low error and a low uncertainty score so this allows us to see that the use of this score as a confidence score make sense.

The potential applications of this approach to business range from being able to recommend measures to avoid overdraft, create shortcuts to most common transactions, improve financial health or find unusual movements in the transaction history. We could also provide the client feedback about how certain our prediction is when it comes to future occurrence.

This research was synthesized in an article accepted for publication that will be presented in the ECML-PKDD 2018 and that is opened in the e-Print service arXiv.