Emergence of Grounded Compositional Language in Multi-Agent Populations
By Igor Mordatch, Pieter Abbeel. 2017
815 citations recorded by the source corpus.
By capturing statistical patterns in large corpora, machine learning has enabled significant advances in natural language processing, including in machine translation, question answering, and sentiment analysis. However, for agents to intelligently interact with humans, simply capturing the statistical patterns is insufficient. In this paper we investigate if, and how, grounded compositional language can emerge as a means to achieve goals in multi-agent populations. Towards this end, we propose a multi-agent learning environment and learning methods that bring about emergence of a basic compositional language. This language is represented as streams of abstract discrete symbols uttered by agents over time, but nonetheless has a coherent structure that possesses a defined vocabulary and syntax. We also observe emergence of non-verbal communication such as pointing and guiding when language communication is unavailable.
Subfields: Artificial Intelligence, Cultural Studies, LLM Agents & Tool Use, Computational Linguistics & Semantics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Percy Liang
- Ruslan Salakhutdinov
- Quoc V. Le
- Kyunghyun Cho
In Cultural Studies
- Jacob Andreas
- Adrian Weller
- Akifumi Wachi
- Stefano V. Albrecht
- Georgios Papoudakis
- Filippos Christianos
- Serhii Havrylov
- Ivan Titov
In LLM Agents & Tool Use
- Shimon Whiteson
- Zongqing Lu
- Jiechuan Jiang
- Stefano V. Albrecht
- Filippos Christianos
- Igor Mordatch
- Jakob N. Foerster
- Georgios Papoudakis
In Computational Linguistics & Semantics
- Jacob Andreas
- Akifumi Wachi
- Stefano V. Albrecht
- Georgios Papoudakis
- Filippos Christianos
- Fei Sha
- Shariq Iqbal
- Adrian Weller
Citations in the corpus
- Counterfactual Multi-Agent Policy Gradients (2017, background): “Also related is work on the emergence of communication between agents, learned by gradient descent (Das et al., 2017; Mordatch & Abbeel, 2017; Lazaridou et al., 2016; Foerster et al., 2016; Sukhbaatar et al., 2016).”
- The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, methodology): “Similar conclusions have been drawn in multi-agent domains: [25] report that multi-agent PG methods such as COMA are outperformed by MADDPG and QMix [27] by a clear margin in the particle-world environment [23] and the StarCraft multi-agent challenge [28].”
- Challenges for Transparency (2017, background): “Exciting work has begun to explore this direction, looking for ways to enable multiple agents to cooperate effectively (Evtimova et al., 2017; Mordatch and Abbeel, 2017).”
- Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication (2020, neutral): “[16] utilizes a communication module to generate discrete communication features for the final policy layer that outputs agents’ actions.”
- Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments (2017, methodology): “Unlike [24], we do not assume that all agents have identical action and observation spaces, or act according to the same policy π .”
- Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games (2017, background): “As the flourishes of deep reinforcement learning (DRL) (Mnih et al. 2015; Silver et al. 2016), researchers start to shed light on tackling multiagent problems (Schmidhuber 1996) with the enhanced learning capabilities, e.g., (Sukhbaatar, Fergus, and others 2016; Mordatch and Abbeel 2017).”
- Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning (2017, background): “The more recent resurgence has focused on deep RL [1, 8, 10, 13, 18, 19, 23, 26].”
- Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks (2020, methodology): “The Multi-Agent Particle Environments (MPE) [Mordatch and Abbeel, 2017] consists of several two-dimensional navigation tasks.”
- Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “In addition, communication has also been shown to emerge from multi-agent RL [31, 32, 33, 34].”
- Learning with Opponent-Learning Awareness (2017, background): “However, most of this recent work considers fully cooperative settings [13, 14, 34] and emergent communication in particular [11, 12, 22, 32, 40].”
- Actor-Attention-Critic for Multi-Agent Reinforcement Learning (2018, methodology): “The two environments are implemented in the multi-agent particle environment framework 1 introduced by Mordatch & Abbeel (2017), and extended by Lowe et al. (2017).”
- Deep Reinforcement Learning for Swarm Systems (2018, methodology): “They learn an encoder-decoder embedding of agent policies via imitation learning based on interactions and evaluate it on a cooperative particle world [Mordatch and Abbeel, 2018] and a competitive two-agent robo sumo environment [Al-Shedivat et al., 2018].”
- Deal or No Deal? End-to-End Learning of Negotiation Dialogues (2017, neutral): “RL has also been used to allow agents to invent new languages (Das et al., 2017; Mordatch and Abbeel, 2017).”
- Deep Communicating Agents for Abstractive Summarization (2018, background): “Compared to prior multi-agent works on logic puzzles (Foerster et al., 2017), language learning (Lazaridou et al., 2016; Mordatch and Abbeel, 2017) and starcraft games (Vinyals et al., 2017), we present the first study in using this framework for long text generation.”
- Learning Attentional Communication for Multi-Agent Cooperation (2018, methodology): “Multi-agent communication in terms of sequences of discrete symbols are investigated [17] and [6].”
- Human-level performance in 3D multiplayer games with population-based reinforcement learning (2018, background): “In contrast to previous work (18, 41, 42, 47, 48, 53, 58, 63, 64), agents do not have access to models of the environment, other agents, or human policy priors, nor can they communicate with each other outside of the game environment.”
- A survey and critique of multiagent deep reinforcement learning (2018, background): “Similarly, Mordatch and Abbeel [160] investigated the emergence of language with the difference that in their setting there were no explicit roles for the agents (i.”
- Emergent Coordination Through Competition (2019, background): “Coordination and cooperation has been studied recently in deepRL in, for example, Lowe et al. (2017); Foerster et al. (2018; 2016); Sukhbaatar et al. (2016); Mordatch & Abbeel (2018), but all of these require some degree of centralization.”