An MDP-Based Recommender System

By Guy Shani, David E. Heckerman, Ronen I. Brafman. 2002

1,059 citations recorded by the source corpus.

Typical Recommender systems adopt a static view of the recommendation process and treat it as a prediction problem. We argue that it is more appropriate to view the problem of generating recommendations as a sequential decision problem and, consequently, that Markov decision processes (MDP) provide a more appropriate model for Recommender systems. MDPs introduce two benefits: they take into account the long-term effects of each recommendation, and they take into account the expected value of each recommendation. To succeed in practice, an MDP-based Recommender system must employ a strong initial model; and the bulk of this paper is concerned with the generation of such a model. In particular, we suggest the use of an n-gram predictive model for generating the initial MDP. Our n-gram model induces a Markovchain model of user behavior whose predictive accuracy is greater than that of existing predictive models. We describe our predictive model in detail and evaluate its performance on real data. In addition, we show how the model can be used in an MDP-based Recommender system.

Subfields: Artificial Intelligence, Information Systems, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Reinforcement Learning

Citations in the corpus

Session-based Recommendations with Recurrent Neural Networks (2015, neutral): “A somewhat different approach to session-based recommendation are Markov Decision Processes (MDPs) (Shani et al., 2002).”
DRN: A Deep Reinforcement Learning Framework for News Recommendation (2018, methodology): “Some recent attempts using reinforcement learning in recommendation either do not model the future reward explicitly (MAB-based works [ 23, 43]), or use discrete user log to represent state and hence can not be scaled to large systems (MDP-based works [35, 36]).”
Deep Q-learning From Demonstrations (2017, neutral): “The learning framework that we have presented in this paper is one that is very common in real world problems such as controlling data centers, autonomous vehicles (Hester and Stone 2013), or recommendation systems (Shani, Heckerman, and Brafman 2005).”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “Reinforcement learning (RL) algorithms, when combined with high-capacity deep neural net function approximators, have shown promise in domains ranging from robotic manipulation [1] to recommender systems [2].”
Learning from Demonstrations for Real World Reinforcement Learning (2017, methodology): “It still remains difficult to apply these algorithms to real world settings such as data centers, autonomous vehicles Hester & Stone (2013), helicopters Abbeel et al. (2007), or recommendation systems Shani et al. (2005).”
Partially Observable Markov Decision Process for Recommender Systems (2016, methodology): “On one hand, the Markovian assumption has been proven to be valid for the sequential recommendations [24].”
STAMP: Short-Term Attention/Memory Priority Model for Session-based Recommendation (2018, background): “The third approach is the Markov chain (MC) based models[3, 15], which utilize sequential connections between the user actions to make prediction.”
Collaborative Filtering with Recurrent Neural Networks (2016, background): “defended the view of recommendation systems as a Markov decision process, and although the predictive model was not their main focus, they did present in [16] a Markov chain approach, improved by some heuristics such as skipping and clustering.”
Collaborative Filtering Recommender Systems (2011, methodology): “Recommenders have been built using Bayesian networks [30, 157], Markov decision processes [136], and neural networks [125].”
Reinforcement Learning based Recommender Systems: A Survey (2021, methodology): “Slate-MDP [112] tries to solve this problem by searching the policy space for each slot in the slate individually.”
Global Context Enhanced Graph Neural Networks for Session-based Recommendation (2020, neutral): “, similarity-based [11] and chain-based [12].”
Learning and adaptivity in interactive recommender systems (2007, background): “In [17], the authors model the states as the set of products previously bought by a user and the system actions correspond to the next possible product recommendations.”
Long and Short-Term Recommendations with Recurrent Neural Networks (2017, background): “defended the view of recommendation systems as a Markov decision process, and although the predictive model was not their main focus, they did present in [18] a Markov chain approach, improved by some heuristics such as skipping and clustering.”
Parallel Recurrent Neural Network Architectures for Feature-rich Session-based Recommendations (2016, background): “Markov Decision Processes (MDP) [25] are the only other approach that aims to provide recommendations in a session-based manner while taking into account information beyond the last click.”
Evaluating Recommendation Systems (2011, background): “In such cases the recommendation engine can be judged by the revenue that it generates for the website [54].”
A Survey of Accuracy Evaluation Metrics of Recommendation Tasks (2009, background): “When we are interested in novel recommendations, the utility can be the log of the inverse popularity of an item, modeling the amountof new information in a recommended item (Shani et al., 2005), but this ignores otherasp cts of user-utility such as the diversity of recommendations.”