Multi-Agent Cooperation and the Emergence of (Natural) Language

By Angeliki Lazaridou, Alex Peysakhovich, Marco Baroni. 2016

488 citations recorded by the source corpus.

The current mainstream approach to train natural language systems is to expose them to large amounts of text. This passive learning is problematic if we are interested in developing interactive machines, such as conversational agents. We propose a framework for language learning that relies on multi-agent communication. We study this learning in the context of referential games. In these games, a sender and a receiver see a pair of images. The sender is told one of them is the target and is allowed to send a message from a fixed, arbitrary vocabulary to the receiver. The receiver must rely on this message to identify the target. Thus, the agents develop their own language interactively out of the need to communicate. We show that two networks with simple configurations are able to learn to coordinate in the referential game. We further explore how to make changes to the game environment to cause the "word meanings" induced in the game to better reflect intuitive semantic properties of the images. In addition, we present a simple strategy for grounding the agents' code into natural language. Both of these are necessary steps towards developing machines that are able to communicate with humans productively.

Subfields: Artificial Intelligence, Cultural Studies, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cultural Studies

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Experience Grounds Language (2020, background): “…(Lewis et al., 2017; He et al., 2018), collaboration (Chai et al., 2017), visual disambiguation (Liu and Chai, 2015; Anderson et al., 2018; Lazaridou et al., 2017), providing emotional support (Rashkin et al., 2019), all of which require inferring mental states and social outcomes—a key…”
Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering (2019, background): “This can also be seen as an instance of two agents cooperating via communication to solve a task (Lazaridou et al., 2017; Lee et al., 2018; Cao et al., 2018).”
Counterfactual Multi-Agent Policy Gradients (2017, background): “Also related is work on the emergence of communication between agents, learned by gradient descent (Das et al., 2017; Mordatch & Abbeel, 2017; Lazaridou et al., 2016; Foerster et al., 2016; Sukhbaatar et al., 2016).”
Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, neutral): “…Barzilay, McCullough, Rambow, DeCristofaro, Korelsky, and Lavoie, 1998; Chai, Fang, Liu, and She (2016); Dodson, Mattei, and Goldsmith, 2011; Henrion and Druzdzel (1990ab), Lazaridou, Peysakhovich, and Baroni, 2017; Neihaus and Young, 2014; Pace and Rosner, 2014; Papamichail, and French, 2003).”
Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments (2017, background): “[16]) may not generalize to more complex tasks.”
Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning (2017, background): “The more recent resurgence has focused on deep RL [1, 8, 10, 13, 18, 19, 23, 26].”
Emergence of Grounded Compositional Language in Multi-Agent Populations (2017, background): “The recent work that is most similar to ours is the application of reinforcement learning approaches towards the purposes of learning a communication protocol, as exemplified by (Foerster et al., 2016; Sukhbaatar et al., 2016; Lazaridou et al., 2016a).”
Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence (2016, background): “The main diﬀerence to our paper consist of (1) that we learn the representations of the images while Lazaridou et al. use a pretrained classiﬁer trained to recognise the categories of objects present in the images, and (2) that we are able to train the agents to hold a multi-step dialogue in…”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “…real-world phenomena, including the emergence of tool usage (Baker et al., 2020), communication (Foerster et al., 2016; Sukhbaatar et al., 2016; Lazaridou et al., 2017; Das et al., 2019), social inﬂuence (Jaques et al., 2019), and inequity aversion (Hughes et al., 2018) can also be better…”
The Hanabi Challenge: A New Frontier for AI Research (2019, background): “e an arbitrary or structured communication channel, and agents must learn to communicate to solve a cooperative problem. Algorithms have been developed to learn to solve riddles and referential games [6, 73], gridworld games requiring coordination [74], object identication via question-and-answer di”
Learning with Opponent-Learning Awareness (2017, background): “However, most of this recent work considers fully cooperative settings [13, 14, 34] and emergent communication in particular [11, 12, 22, 32, 40].”
A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning (2017, methodology): “In future work, we will consider maintaining diversity among oracles via loss penalties based on policy dissimilarity, general response graph topologies, environments such as emergent language games [58] and RTS games [96, 84], and other architectures for prediction of behavior, such as opponent mod”
A survey and critique of multiagent deep reinforcement learning (2018, background): “• Learning communication [161, 160, 162, 163, 164, 165].”
ROMA: Multi-Agent Reinforcement Learning with Emergent Roles (2020, background): “Works on the emergence of communication (Foerster et al., 2016; Lazaridou et al., 2017; Das et al., 2017; Mordatch & Abbeel, 2018; Wang et al., 2020b), the emergence of fairness (Jiang & Lu, 2019), and the emergence of tool usage (Baker et al., 2020) provide a deep learning perspective in…”
Emergent Communication through Negotiation (2018, background): “…has mainly studied the emergence of communication in cooperative games like referential games, variants of the Lewis signaling game (Lewis, 1969), where messages are used to disambiguate between different possible referents (Goldman et al., 2007; Lazaridou et al., 2016; Evtimova et al., 2017).”
"Other-Play" for Zero-Shot Coordination (2020, background): “This is particularly important in applications such as cooperative game playing, communication, or autonomous driving (Foerster et al., 2016; Lazaridou et al., 2016; Sukhbaatar et al., 2016; Resnick et al., 2018).”
Semantic Communications for Future Internet: Fundamentals, Applications, and Challenges (2022, unknown): “However, most of the works merely focus on some simple and specific AI tasks such as image-related referential games [116], where the accomplishment of SE may be spurious owing to the inscrutable patterns of the single transmitted objects [117].”