Deep Recurrent Q-Learning for Partially Observable MDPs

By Matthew J. Hausknecht, Peter Stone. 2015

1,965 citations recorded by the source corpus.

Deep Reinforcement Learning has yielded proficient controllers for complex tasks. However, these controllers have limited memory and rely on being able to perceive the complete game screen at each decision point. To address these shortcomings, this article investigates the effects of adding recurrency to a Deep Q-Network (DQN) by replacing the first post-convolutional fully-connected layer with a recurrent LSTM. The resulting \textit{Deep Recurrent Q-Network} (DRQN), although capable of seeing only a single frame at each timestep, successfully integrates information through time and replicates DQN's performance on standard Atari games and partially observed equivalents featuring flickering game screens. Additionally, when trained with partial observations and evaluated with incrementally more complete observations, DRQN's performance scales as a function of observability. Conversely, when trained with full observations and evaluated with partial observations, DRQN's performance degrades less than DQN's. Thus, given the same length of history, recurrency is a viable alternative to stacking a history of frames in the DQN's input layer and while recurrency confers no systematic advantage when learning to play the game, the recurrent net can better adapt at evaluation time if the quality of observations changes.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Reinforcement Learning

Citations in the corpus

Memory-based control with recurrent neural networks (2015, methodology): “Several groups [15, 1, 5] have studied the use of model-free algorithms with recurrent networks to solve POMDPs with discrete action spaces.”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, methodology): “Following the training strategy used in the recurrent DQN literature (Hausknecht and Stone, 2015; Yuan et al., 2018), we use the ﬁrst 2 transitions to estimate the recurrent states, and the last 2 transitions for updating the model parameters.”
Value Prediction Network (2017, background): “A lot of variations of DQN have been proposed for learning better state representation [37, 16, 9, 22, 36, 24], including the use of memory-based networks for handling partial observability [9, 22, 24], estimating both state-values and advantage-values as a decomposition of Q-values [37], learning s”
Neural Approaches to Conversational AI (2018, neutral): “in the case of Deep Q-Network (DQN), Q(s;a;) takes the form of deep neural networks, such as multi-layer perceptrons and convolutional networks (Tesauro, 1995; Mnih et al., 2015), recurrent network (Hausknecht and Stone, 2015; Li et al., 2015), etc. More examples will be seen in later chapters. Furt”
Counterfactual Multi-Agent Policy Gradients (2017, methodology): “To condition on this full history, a deep RL agent may make use of a recurrent neural network (Hausknecht & Stone, 2015), typically making use of a gated model such as LSTM (Hochreiter & Schmidhuber, 1997) or GRU (Chung et al., 2014).”
Learning to Communicate with Deep Multi-Agent Reinforcement Learning (2016, background): “Hausknecht and Stone [17] propose deep recurrent Q-networks (DRQN) to address single-agent, partially observable settings.”
Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, background): “Hausknecht and Stone (2015) propose deep recurrent Q-networks (DRQN) that make use of recurrent neural networks.”
Value-Decomposition Networks For Cooperative Multi-Agent Learning (2017, methodology): “Since we consider partially observed environments our Q -functions are deﬁned over agent observation histories, Q ( h t , a t ) , and we incorporate a recurrent network similarly to Hausknecht and Stone (2015).”
RUDDER: Return Decomposition for Delayed Rewards (2018, methodology): “LSTM was already used in reinforcement learning [112] for advantage learning [4], for constructing a potential function for reward shaping by representing the return by a sum of LSTM outputs across an episode [124], and learning policies [44, 80, 45].”
Fitted Q-Learning for Relational Domains (2020, background): “…related to our work is the recent work on Deep RL. Deep Q networks (DQN), ﬁrst introduced by Mnih et al. (2015) and later extended by others (Hausknecht and Stone 2015; Van Hasselt, Guez, and Silver 2016; Mnih et al. 2016; Anschel, Baram, and Shimkin 2017) can also be seen as ﬁtted…”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, methodology): “DRQN [44] applied a modiﬁcation to the DQN by combining a Long Short Term Memory (LSTM) with a Deep Q-Network.”
Deep Reinforcement Learning framework for Autonomous Driving (2017, methodology): “The application of DRQN in [4] to Atari games does not show the full power of the model, since the MDP assumption is usually enough for Atari games, hence the authors try a variant of the Pong game namely Flickering Pong, where they show the advantage of adding recurrence.”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “recently (Duan et al., 2016). Following the success of DQN (Mnih et al., 2015), revised deep reinforcement learning methods appeared to improve the performance on various of applications in robotics (Hausknecht and Stone, 2015; Wang et al., 2016). Tai and Liu (2016a,c) used DQN to help the robot lea”
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation (2016, background): “There has been some recent work in using recurrent networks in conjunction with reinforcement learning [15, 23].”
Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation (2018, methodology): “ot fully observable state st. LSTM has been verified to improve performance on POMDP by summarizing entire historical observations ct = f (o1,o2,...,ot) when using policy gradient [39] and Q-learning [14]. In LSTM, the hidden units ct is to encapsulate the entire observation history o1:t. In order t”
Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability (2017, methodology): “Using DRQNs, our approach extends the single-agent, fully-observable MTRL method proposed by Rusu et al. (2015) to Dec-POMDP MT-MARL.”
Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning (2017, methodology): “In deep RL, this is accomplished by modelling the Q -function with a recurrent neural network (Hausknecht & Stone, 2015), utilising a gated architecture such as LSTM (Hochreiter & Schmid-huber, 1997) or GRU (Chung et al., 2014).”