Recurrent Reinforcement Learning: A Hybrid Approach

By Xiujun Li, Lihong Li, Jianfeng Gao, Xiaodong He, Jianshu Chen, Li Deng, Ji He. 2015

77 citations recorded by the source corpus.

Successful applications of reinforcement learning in real-world problems often require dealing with partially observable states. It is in general very challenging to construct and infer hidden states as they often depend on the agent's entire interaction history and may require substantial domain knowledge. In this work, we investigate a deep-learning approach to learning the representation of states in partially observable tasks, with minimal prior knowledge of the domain. In particular, we propose a new family of hybrid models that combines the strength of both supervised learning (SL) and reinforcement learning (RL), trained in a joint fashion: The SL component can be a recurrent neural networks (RNN) or its long short-term memory (LSTM) version, which is equipped with the desired property of being able to capture long-term dependency on history, thus providing an effective way of learning the representation of hidden states. The RL component is a deep Q-network (DQN) that learns to optimize the control for maximizing long-term rewards. Extensive experiments in a direct mailing campaign problem demonstrate the effectiveness and advantages of the proposed approach, which performs the best among a set of previous state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Networks and Communications, Reinforcement Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Reinforcement Learning

In Model Architectures

Citations in the corpus

Reinforcement Learning with Unsupervised Auxiliary Tasks (2016, background): “Although learning environment models as auxiliary tasks could improve RL agents (e.g. Lin & Mitchell (1992); Li et al. (2015)), this has not yet been shown to work in rich visual environments.”
Neural Approaches to Conversational AI (2018, neutral): “k (DQN), Q(s;a;) takes the form of deep neural networks, such as multi-layer perceptrons and convolutional networks (Tesauro, 1995; Mnih et al., 2015), recurrent network (Hausknecht and Stone, 2015; Li et al., 2015), etc. More examples will be seen in later chapters. Furthermore, it is possible to r”
Causally Correct Partial Models for Reinforcement Learning (2020, neutral): “…next-step models (Oh et al., 2015; Ha & Schmidhuber, 2018; Chiappa et al., 2017; Schmidhuber, 2010; Xie et al., 2016; Deisenroth & Rasmussen, 2011; Lin & Mitchell, 1992; Li et al., 2015; Diuk et al., 2008; Igl et al., 2018; Ebert et al., 2018; Kaiser et al., 2019; Janner et al., 2019).”
Shaping Belief States with Generative Environment Models for RL (2019, background): “The idea of learning general world models to support decision-making is probably one of the most pervasive ideas in AI research, [30, 11, 41, 42, 14, 43, 44, 45, 46, 33, 30, 47, 48, 3, 49, 50, 51, 52].”
Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (2015, methodology): “The policy π train is generated by training a recurrent neural network on the original data to ﬁt a Q-value function (Li et al., 2015b).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “A common approach to enable learning about more than a specific task is to augment the loss function with auxiliary losses to provide denser training signals in RL (Li et al., 2015; Lample and Chaplot, 2017).”
Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks (2016, background): “Recurrent DQN was also successful in the email campaign challenge (Li et al., 2015).”
Learning to Navigate in Complex Environments (2016, background): “Speciﬁcally, Li et al. (2016) consider a supervised loss for ﬁtting a recurrent model on the hidden representations to predict the next observed state, in the context of imitation learning of sequences provided by experts, and Lample & Chaplot (2016) show that the performance of a DQN agent in a…”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “A different form of transfer can be utilised to help RL in the form of multitask training [66, 43, 69].”
An Introduction to Deep Reinforcement Learning (2018, unknown): “Note that learning the model can share the hidden-state representations with a value-based approach by sharing neural network parameters (Li et al., 2015).”
Deep Reinforcement Learning: An Overview (2017, background): “Li et al. (2015) also attempted to maximize lifetime value of customers.”
Adaptive Auxiliary Task Weighting for Reinforcement Learning (2019, background): “While reinforcement learning with low-dimensional state input can also benefit from auxiliary tasks in partially observable environments [17, 18], auxiliary tasks have been more commonly used for reinforcement learning from images or other high-dimensional sensor readings.”
Robust Actor-Critic Contextual Bandit for Mobile Health (mHealth) Interventions (2018, methodology): “a challenging problem to reliably evaluate the sequential decision making (e.g., bandit and reinforcement learning) algorithms when the simulator (like the Atari games) is unavailable (Li et al. 2010;Li et al. 2015). After the model is trained, we use it to interact with the enviroment (or simulator”
Deep Reinforcement Learning with an Unbounded Action Space (2015, methodology): “Li et al. (2015) developed a joint training approach for recurrent reinforcement learning and demonstrate its effectiveness on a customer relationship management task.”
Deep Q-Networks for Accelerating the Training of Deep Neural Networks (2016, background): “Although there is no reason to believe that such strategies are optimal, reinforcement learning with long-term predictions are inherently difficult (Li et al., 2015), especially when we are using a model-free method and each episode would take a day to play.”