Multi-focus Attention Network for Efficient Deep Reinforcement Learning

By Jinyoung Choi, Beom-Jin Lee, Byoung-Tak Zhang. 2017

52 citations recorded by the source corpus.

Deep reinforcement learning (DRL) has shown incredible performance in learning various tasks to the human level. However, unlike human perception, current DRL models connect the entire low-level sensory input to the state-action values rather than exploiting the relationship between and among entities that constitute the sensory input. Because of this difference, DRL needs vast amount of experience samples to learn. In this paper, we propose a Multi-focus Attention Network (MANet) which mimics human ability to spatially abstract the low-level sensory input into multiple entities and attend to them simultaneously. The proposed method first divides the low-level input into several segments which we refer to as partial states. After this segmentation, parallel attention layers attend to the partial states relevant to solving the task. Our model estimates state-action values using these attended partial states. In our experiments, MANet attains highest scores with significantly less experience samples. Additionally, the model shows higher performance compared to the Deep Q-network and the single attention model as benchmarks. Furthermore, we extend our model to attentive communication model for performing multi-agent cooperative tasks. In multi-agent cooperative task experiments, our model shows 20% faster learning than existing state-of-the-art model.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Towards Interpretable Reinforcement Learning Using Attention Augmented Agents (2019, background): “The first agent uses the same attention mechanism except that the queries are a learnable bias tensor which does not depend on the LSTM state (this style of attention is similar to the one used in [26], although that model does not include many of the adaptations used here).”
Actor-Attention-Critic for Multi-Agent Reinforcement Learning (2018, background): “…approaches that propose an actor-critic framework consisting of centralized training with decentralized execution (Lowe et al., 2017; Foerster et al., 2017a), as well as some approaches that utilize attention in a fully centralized multi-agent setting (Choi et al., 2017; Jiang & Lu, 2018).”
Reinforcement Learning with Attention that Works: A Self-Supervised Approach (2019, background): “Different domains have previously shown to be better addressed with different, sometimes conflicting implementation choices [28,23,4].”
Parallel Knowledge Transfer in Multi-Agent Reinforcement Learning (2020, background): “Motivated by attention [22] introduced for information extraction in deep learning, attention mechanism has recently emerged in reinforcement learning framework [23], [24].”
A review of cooperative multi-agent deep reinforcement learning (2019, methodology): “The main idea in this work is to use an attention mechanism [23, 53] to select relevant information for each agent during the training.”
Neuroevolution of self-interpretable agents (2020, background): “tive to apply attention directly to individual pixels, and we rather operate on image patches (which have lower dimensions) instead. Although not in the context of self-attention, previous work (e.g. [10, 13, 22, 24, 31, 95]) segments the visual input and attend to the patches rather than individual”
Free-Lunch Saliency via Attention in Atari Agents (2019, methodology): “[10] DQN GridWorld 0 Custom 3 Soft key-value See [10] 3 Raw Reward”
Goal-Aware Cross-Entropy for Multi-Target Reinforcement Learning (2021, unknown): “There are also attention-based methods for reinforcement learning which use inputs from different parts of the state [4, 9].”
Attention in Natural Language Processing (2019, background): “recommendation [22], [23], time-series analysis [24], [25], games [26], and mathematical problems [27], [28].”