Learning Multiagent Communication with Backpropagation
By Sainbayar Sukhbaatar, Arthur Szlam, Rob Fergus. 2016
1,366 citations recorded by the source corpus.
Many tasks in AI require the collaboration of multiple agents. Typically, the communication protocol between agents is manually specified and not altered during training. In this paper we explore a simple neural model, called CommNet, that uses continuous communication for fully cooperative tasks. The model consists of multiple agents and the communication between them is learned alongside their policy. We apply this model to a diverse set of tasks, demonstrating the ability of the agents to learn to communicate amongst themselves, yielding improved performance over non-communicative agents and baselines. In some cases, it is possible to interpret the language devised by the agents, revealing simple but effective strategies for solving the task at hand.
Subfields: Artificial Intelligence, Natural Language Processing, LLM Agents & Tool Use
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Mohit Bansal
- Jianfeng Gao
- Ruslan Salakhutdinov
- Quoc V. Le
- Percy Liang
- Kyunghyun Cho
In Natural Language Processing
- Mohit Bansal
- Kyunghyun Cho
- Jianfeng Gao
- Jason Aaron Edward Weston
- Yejin Choi
- Yoshua Bengio
- Christopher D. Manning
- Richard Socher
In LLM Agents & Tool Use
- Shimon Whiteson
- Zongqing Lu
- Chongjie Zhang
- Jiechuan Jiang
- Stefano V. Albrecht
- Filippos Christianos
- Richard Ryan Williams
- Igor Mordatch
Citations in the corpus
- VAIN: Attentional Multi-agent Predictive Modeling (2017, positive): “Two related approaches that learn multi-agent interactions on a graph structure are: Interaction Networks [9] which learn a physical simulation of objects that exhibit binary relations and Communication Networks (CommNets) [11], presented for learning optimal communications between agents.”
- Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs (2016, background): “This work remained practically unnoticed and has been rediscovered only recently [24, 37].”
- Counterfactual Multi-Agent Policy Gradients (2017, background): “Also related is work on the emergence of communication between agents, learned by gradient descent (Das et al., 2017; Mordatch & Abbeel, 2017; Lazaridou et al., 2016; Foerster et al., 2016; Sukhbaatar et al., 2016).”
- Learning to Communicate with Deep Multi-Agent Reinforcement Learning (2016, background): “Unlike the recent work in [10], we consider discrete communication channels.”
- Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, background): “CommNet (Sukhbaatar et al., 2016) use a centralised network architecture to exchange information between agents.”
- Value-Decomposition Networks For Cooperative Multi-Agent Learning (2017, background): “Further, in the context of the introduced agent, we evaluate weight sharing, role information and information channels as additional enhancements that have recently been reported to improve sample complexity and memory requirements (Hausknecht, 2016; Foerster et al., 2016; Sukhbaatar et al., 2016).”
- Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication (2020, positive): “Following [19], we compare the average vector norm of messages sent by agents at each location.”
- Relational inductive biases, deep learning, and graph networks (2018, methodology): “Various models, including CommNet (Sukhbaatar et al., 2016), structure2vec (Dai et al., 2016) (in the version of (Dai et al., 2017)), and Gated Graph Sequence Neural Networks (Li et al., 2016) have used a φ e which does not directly compute pairwise interactions, but instead ignore the receiver…”
- Geometric Matrix Completion with Recurrent Multi-Graph Neural Networks (2017, neutral): “2005; 2009 (see more recent formulation (Li et al., 2015; Sukhbaatar et al., 2016)).”
- Geometric Deep Learning: Going beyond Euclidean data (2016, background): “Recent works [55], [50], [51], [79], [80] relax the requirements of approaching the steady state or using repeated applications of the same Cθ.”
- Attention-based Graph Neural Network for Semi-supervised Learning (2018, background): “We give a typical example of a GNN in Section 3, but several diverse variations have been proposed in [10, 17, 29, 24, 39, 8, 15, 13, 3, 32, 38, 23, 37, 38].”
- Learning Multi-Arm Manipulation Through Collaborative Teleoperation (2020, background): “Multi-Agent Reinforcement Learning: Multi-Agent Re-inforcement Learning [5, 35] in cooperative settings has been widely studied [11, 12, 14, 21, 26, 34], and applied to domains such as video games [28] and visual question answering [9].”
- Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments (2017, background): “icies), (4) we learn continuous policies whereas they learn discrete policies. 2 Recent work has focused on learning grounded cooperative communication protocols between agents to solve various tasks [31, 8, 25]. However, these methods are usually only applicable when the communication between agent”
- Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games (2017, methodology): “29, 30]. One of the key components in using DRL is to learn a communication protocol among agents. Representative solutions include the differentiable inter-agent learning (DIAL) [19] and the CommNet [20], both of which are end-to-end trainable by back-propagation. DIAL [19] was introduced in partia”
- Emergence of Grounded Compositional Language in Multi-Agent Populations (2017, methodology): “A similar approach was employed by (Foerster et al., 2016; Sukhbaatar et al., 2016) to compute gradients for communication actions, although the latter still employed modelfree methods for physical action computation.”
- Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning (2017, background): “Sukhbaatar et al. (2016) also show that it is possible to learn to communicate by backpropagation.”
- Multi-Agent Cooperation and the Emergence of (Natural) Language (2016, background): “Most re-lated to our work, Sukhbaatar et al. (2016) and Foerster et al. (2016) show that neural networks can evolve communication in the context of games without a pre-coded protocol.”