Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks

By Jakob N. Foerster, Yannis Assael, Nando de Freitas, Shimon Whiteson. 2016

156 citations recorded by the source corpus.

We propose deep distributed recurrent Q-networks (DDRQN), which enable teams of agents to learn to solve communication-based coordination tasks. In these tasks, the agents are not given any pre-designed communication protocol. Therefore, in order to successfully communicate, they must first automatically develop and agree upon their own communication protocol. We present empirical results on two multi-agent learning problems based on well-known riddles, demonstrating that DDRQN can successfully solve such tasks and discover elegant communication protocols to do so. To our knowledge, this is the first time deep reinforcement learning has succeeded in learning communication protocols. In addition, we present ablation experiments that confirm that each of the main components of the DDRQN architecture are critical to its success.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Learning to Communicate with Deep Multi-Agent Reinforcement Learning (2016, background): “This paper extends our own work in [3], which proposes a variation of DQN for protocol learning.”
Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability (2017, methodology): “Foerster et al. (2016) present architectures to learn communication protocols for Dec-POMDP RL, noting best performance using a centralized approach with inter-agent backpropagation and parameter sharing.”
Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence (2016, background): “In [2, 3] the authors solve puzzles with multiple agents with one-bit communication.”
Multi-Agent Cooperation and the Emergence of (Natural) Language (2016, background): “Most re-lated to our work, Sukhbaatar et al. (2016) and Foerster et al. (2016) show that neural networks can evolve communication in the context of games without a pre-coded protocol.”
Learning Multiagent Communication with Backpropagation (2016, methodology): “n contrast to these approaches, our model uses a deep network for both agent control and communication. From a MARL perspective, the closest approach to ours is the concurrent work of Foerster et al. [5]. This also uses a deep reinforcement learning in multi-agent partially observable tasks, speciﬁc”
Multiagent cooperation and competition with deep reinforcement learning (2015, background): “A future direction of the present approach is to study of how communication codes [27, 28] and consensus [28–30] can emerge between interacting agents in complex environments without any a priori agreements, rules, or even high-level concepts of themselves and their environment.”
Decentralized Likelihood Quantile Networks for Improving Performance in Deep Multi-Agent Reinforcement Learning (2018, background): “Earlier attempts disabled experience reply due to non-concurrent evolving across agents’ policies (Foerster et al., 2016).”
Opponent Modeling in Deep Reinforcement Learning (2016, neutral): “Foerster et al. (2016) use modern neural networks to learn a group of parameter-sharing agents that solve a coordination task, where each agent is controlled by a deep recurrent Q-Network (Hausknecht & Stone, 2015).”
Playing FPS Games with Deep Reinforcement Learning (2016, background): “Foerster et al. (2016) consider a multi-agent scenario where they use deep distributed recurrent neural networks to communicate between different agent in order to solve riddles.”
Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning (2018, background): “This clearly demonstrates the importance of learning conventions for successful gameplay in Hanabi: Roughly 40% of the information is obtained through conventions rather than through the grounded information and card counting. et al. (2016b).”
Deep Variational Reinforcement Learning for POMDPs (2018, methodology): “While previous work often used Q-learning to train the policy (Hausknecht & Stone, 2015; Zhu et al., 2017; Foerster et al., 2016; Narasimhan et al., 2015), we use n-step A2C.”
Modeling Others using Oneself in Multi-Agent Reinforcement Learning (2018, background): “A large body of work in deep multi-agent RL focuses on partially visible, fully cooperative settings (Foerster et al., 2016a;b; Omidshaﬁei et al., 2017) and emergent communication (Lazaridou et al., 2016; Foerster et al., 2016a; Sukhbaatar et al., 2016; Das et al., 2017; Mordatch & Abbeel, 2017)…”
A Paradigm for Situated and Goal-Driven Language Learning (2016, background): “, 2016; Weston, 2016) and multi-agent communication (Lazaridou et al., 2016; Andreas and Klein, 2016; Foerster et al., 2016; Sukhbaatar et al., 2016) can be fit into the paradigm described so far.”
Multiagent Learning and Coordination with Clustered Deep Q-Network (2019, background): “Centralized training of decentralized policies is a standard approach of Multiagent Reinforcement Learning [5] and Deep Reinforcement Learning [3] (DRL).”
Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks (2018, background): “The work by Foerster et al. (2016b) also learns a communication protocol where agents communicate in a discrete manner through their actions.”
Toward a Brain-Inspired System: Deep Recurrent Reinforcement Learning for a Simulated Self-Driving Agent (2019, background): “Also, Foerster et al. (2016) proposed to use multi-agent to describe a state distributively.”
Cooperation and communication in multiagent deep reinforcement learning (2016, background): “Second, in multiagent settings, Foerster et al. (2016a) show that the 21 replay memory didn’t help with learning.”
Stochastic Double Deep Q-Network (2019, methodology): “Following the DQN, many new DRL algorithms have emerged, such as double DQN (DDQN) [6], dueling DQN [7], deep recurrent Q-network [8], [9], deep deterministic policy gradient [10], prioritized experience replay [11], asynchronous advantage Actor-Critic [12], deep primal-dual policy learning [13], et”