Deep Attention Recurrent Q-Network

By Ivan Sorokin, Alexey Seleznev, Mikhail Pavlov, Aleksandr Fedorov, Anastasiia Ignateva. 2015

170 citations recorded by the source corpus.

A deep learning approach to reinforcement learning led to a general learner able to train on visual input to play a variety of arcade games at the human and superhuman levels. Its creators at the Google DeepMind's team called the approach: Deep Q-Network (DQN). We present an extension of DQN by "soft" and "hard" attention mechanisms. Tests of the proposed Deep Attention Recurrent Q-Network (DARQN) algorithm on multiple Atari 2600 games show level of performance superior to that of DQN. Moreover, built-in attention mechanisms allow a direct online monitoring of the training process by highlighting the regions of the game screen the agent is focusing on when making decisions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Towards Interpretable Reinforcement Learning Using Attention Augmented Agents (2019, background): “image classification and captioning [12, 13, 14, 15, 16, 17, 18, 19, 20], text classification [21, 22], generative models [6, 23, 24], object tracking [25], and reinforcement learning [26, 27, 28].”
Deep Reinforcement Learning framework for Autonomous Driving (2017, methodology): “This integration was performed in [16].”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “However, the DQN outperformed the DRQN and DARQN on games requiring quick reactions, where Q-values can fluctuate more rapidly.”
Memory-Efficient Backpropagation Through Time (2016, background): “LSTM [9] in particular is an RNN architecture that has excelled in sequence generation [3, 13, 4], speech recognition [5] and reinforcement learning [12, 10] settings.”
Learn to Interpret Atari Agents (2018, methodology): “[34] incorporate the hard and soft attention mechanisms [35] used for image caption generation into deep recurrent Q-network [36], in which the policy layers in the original DQN are replaced with a recurrent neural network.”
An Atari Model Zoo for Analyzing, Visualizing, and Comparing Deep Reinforcement Learning Agents (2018, background): “Beyond algorithms, there are many alternate architectures that might have interesting effects on representation and decision-making, for example recurrent architectures [14], or architectures that exploit attention [47].”
Multi-focus Attention Network for Efficient Deep Reinforcement Learning (2017, methodology): “Recently, to overcome this limitation, works like (Sorokin et al. 2015; Oh et al. 2016) applied attention concepts to the DRL models in order to learn Atari 2600 and Minecraft maze task, respectively.”
Reinforcement Learning with Attention that Works: A Self-Supervised Approach (2019, methodology): “Different domains have previously shown to be better addressed with different, sometimes conflicting implementation choices [28,23,4].”
Neuroevolution of self-interpretable agents (2020, methodology): “and only the winners can access the input and also other members’ states. This modular mechanism improved generalization on Atari. Attention is also explicitly targeted for interpretability in RL. In [88], the authors incorporated soft and hard attention mechanism into the deep recurrent Q-network, ”
Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications (2018, unknown): “As a result, DARQN achieves a score of 7263 compared with 1284 and 1421 of DQN and DRQN on the game “Seaquest,” respectively.”
Free-Lunch Saliency via Attention in Atari Agents (2019, methodology): “At time t, input frames st are turned into embeddings vt, ht is the recurrent state: (a) Sparse convolutional block; (b) Dense convolutional block; (c) Region-Sensitive Module [42]; (d) DARQN [36] (N = 256) and [9] (N = 512); (e) architecture from [26]; (f) our architecture (ht ≡ 0).”
End-to-End Deep Reinforcement Learning for Lane Keeping Assist (2016, methodology): “The idea is exploited in [16] to improve the DQN and DRQN models, by including a filter after the CNN features extractor, resulting in the DARQN model.”