Learning to Communicate with Deep Multi-Agent Reinforcement Learning

By Jakob N. Foerster, Yannis Assael, Nando de Freitas, Shimon Whiteson. Neural Information Processing Systems, 2016

1,917 citations recorded by the source corpus.

We consider the problem of multiple agents sensing and acting in environments with the goal of maximising their shared utility. In these environments, agents must learn communication protocols in order to share information that is needed to solve the tasks. By embracing deep neural networks, we are able to demonstrate end-to-end learning of protocols in complex environments inspired by communication riddles and multi-agent computer vision problems with partial observability. We propose two approaches for learning in these domains: Reinforced Inter-Agent Learning (RIAL) and Differentiable Inter-Agent Learning (DIAL). The former uses deep Q-learning, while the latter exploits the fact that, during learning, agents can backpropagate error derivatives through (noisy) communication channels. Hence, this approach uses centralised learning but decentralised execution. Our experiments introduce new environments for studying the learning of communication protocols and present a set of engineering innovations that are essential for success in these domains.

Subfields: Artificial Intelligence, Reinforcement Learning

View Paper: https://arxiv.org/abs/1605.06676

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Counterfactual Multi-Agent Policy Gradients (2017, methodology): “This centralised training of decentralised policies is a standard paradigm for multi-agent planning (Oliehoek, Spaan, and Vlassis 2008; Kraemer and Banerjee 2016) and has recently been picked up by the deep RL community (Foerster et al. 2016; Jorge, Kågebäck, and Gustavsson 2016).”
Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, methodology): “DIAL (Foerster et al., 2016) utilises an end-to-end diﬀerentiable architecture that allows for a learned inter-agent communication to emerge via backpropagation.”
Value-Decomposition Networks For Cooperative Multi-Agent Learning (2017, background): “Further, in the context of the introduced agent, we evaluate weight sharing, role information and information channels as additional enhancements that have recently been reported to improve sample complexity and memory requirements (Hausknecht, 2016; Foerster et al., 2016; Sukhbaatar et al., 2016).”
Data Augmentation Generative Adversarial Networks (2017, background): “…(Gu et al., 2015), speech recognition (Hinton et al., 2012a), and synthesis (Wang et al., 2017), learning from human play (Clark & Storkey, 2015) and reinforcement learning (Mnih et al., 2015; Silver et al., 2016; Foerster et al., 2016; Van Hasselt et al., 2016; Gu et al., 2016) among others.”
Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication (2020, neutral): “[6] defined discrete symbols as messages.”
Relational inductive biases, deep learning, and graph networks (2018, background): “…et al., 2016), partial tree traversals in a state-action graph (Guez et al., 2018; Farquhar et al., 2018), hierarchical action policies (Andreas et al., 2017), multi-agent communication channels (Foerster et al., 2016), “capsules” (Sabour et al., 2017), and programs (Parisotto et al., 2017).”
Learning Multi-Arm Manipulation Through Collaborative Teleoperation (2020, background): “inforcement Learning [5, 38] in cooperative settings has been widely studied [11, 12, 15, 23, 28, 37], and applied to domains such as video games [31] and visual question answering [9].”
Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments (2017, methodology): “icies), (4) we learn continuous policies whereas they learn discrete policies. 2 Recent work has focused on learning grounded cooperative communication protocols between agents to solve various tasks [31, 8, 25]. However, these methods are usually only applicable when the communication between agent”
Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games (2017, background): “The idea of learning independent agents can also be found (Lauer and Riedmiller 2000; Kapetanakis and Kudenko 2002; Lauer and Riedmiller 2004; Foerster et al. 2016).”
Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning (2017, background): “The more recent resurgence has focused on deep RL [1, 8, 10, 13, 18, 19, 23, 26].”
Emergence of Grounded Compositional Language in Multi-Agent Populations (2017, methodology): “A similar approach was employed by (Foerster et al. 2016; Sukhbaatar, Szlam, and Fergus 2016) to compute gradients for communication actions, although the latter still employed model-free meth-ods for physical action computation.”
Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning (2017, methodology): “To avoid this problem, previous work on deep multi-agent RL has limited the use of experience replay to short, recent buffers (Leibo et al., 2017) or simply disabled replay alto-gether (Foerster et al., 2016).”
Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence (2016, methodology): “In Guess Who? two players are each assigned one character from a set of 24 characters.”
Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, background): “Enjoying these advantages, many advanced deep MARL approaches adopt the PDSP paradigm, including value-based methods [4–8], policy gradients [9–13] and communication learning algorithms [14, 15].”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, methodology): “The framework of centralized training with decentralized execution (CTDE) (Foerster et al., 2016; Gupta et al., 2017; Rashid et al., 2018) combines the advantages of these two paradigms.”
Weighted QMIX: Expanding Monotonic Value Function Factorisation (2020, background): “Exploiting these possibilities can greatly improve the efﬁciency of learning [7, 9].”
Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks (2020, methodology): “These algorithm classes are frequently used in the literature either as baselines or building blocks for more complex algorithms [He et al., 2016, Sukhbaatar et al., 2016, Foerster et al., 2016, Raileanu et al., 2018, Jaques et al., 2019, Iqbal and Sha, 2019, Du et al., 2019, Ryu et al., 2020].”
Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning (2019, methodology): “5 points, tabular IQL does well in this task, matching the BAD results from Foerster et al. (2019). However, just by adding the greedy action as an additional input, we obtain an average performance of 9.”
Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “[32] Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson.”
QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning (2019, methodology): “Other related work includes CommNet (Sukhbaatar et al., 2016), DIAL (Foerster et al., 2016), ATOC (Jiang & Lu, 2018), and SCHEDNET (Kim et al., 2019), which exploit inter-agent communication in execution.”