Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols

By Serhii Havrylov, Ivan Titov. 2017

328 citations recorded by the source corpus.

Learning to communicate through interaction, rather than relying on explicit supervision, is often considered a prerequisite for developing a general AI. We study a setting where two agents engage in playing a referential game and, from scratch, develop a communication protocol necessary to succeed in this game. Unlike previous work, we require that messages they exchange, both at train and test time, are in the form of a language (i.e. sequences of discrete symbols). We compare a reinforcement learning approach and one using a differentiable relaxation (straight-through Gumbel-softmax estimator) and observe that the latter is much faster to converge and it results in more effective protocols. Interestingly, we also observe that the protocol we induce by optimizing the communication success exhibits a degree of compositionality and variability (i.e. the same information can be phrased in different ways), both properties characteristic of natural languages. As the ultimate goal is to ensure that communication is accomplished in natural language, we also perform experiments where we inject prior information about natural language into our model and study properties of the resulting protocol.

Subfields: Artificial Intelligence, Cultural Studies

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cultural Studies

Citations in the corpus

Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning (2017, background): “The more recent resurgence has focused on deep RL [1, 8, 10, 13, 18, 19, 23, 26].”
Learning Attentional Communication for Multi-Agent Cooperation (2018, background): “Multi-agent communication in terms of sequences of discrete symbols are investigated [17] and [6].”
XOC: Explainable Observer-Classifier for Explainable Binary Decisions (2019, background): “Learning to communicate in a multi-agent setting has recently gained interest in the research community mostly due to the emergence of deep reinforcement learning (Foerster et al., 2016; Havrylov & Titov, 2017).”
Emergent Communication through Negotiation (2018, background): “By repeatedly interacting with other agents learning at the same time, agents can gradually bootstrap complex behaviour, including motor skills (Bansal et al., 2017) and linguistic communication (Lazaridou et al., 2016; Havrylov & Titov, 2017).”
Learning to Schedule Communication in Multi-agent Reinforcement Learning (2019, background): “…Lowe et al. (2017); Foerster et al. (2017a); Omidshaﬁei et al. (2017); Foerster et al. (2016); Sukhbaatar et al. (2016); Mordatch & Abbeel (2017); Havrylov & Titov (2017); Palmer et al. (2017); Peng et al. (2017); Fo-erster et al. (2017c); Tampuu et al. (2017); Leibo et al. (2017); Foerster et…”
SEQˆ3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression (2019, methodology): “, 2017) and converges faster than reinforcement learning (Havrylov and Titov, 2017).”
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense (2020, background): “[444] Serhii Havrylov, Ivan Titov, Emergence of language with multi-agent games: Learning to communicate with sequences of symbols, in: Advances in Neural Information Processing Systems (NeurIPS), 2017.”
Consequentialist conditional cooperation in social dilemmas with imperfect information (2017, background): “2017; Peysakhovich and Lerer 2017), and the emergence of language (Lazaridou, Peysakhovich, and Baroni 2017; Das et al. 2017; Evtimova et al. 2017; Havrylov and Titov 2017; Jorge, Kågebäck, and Gustavsson 2016).”
Natural Language Does Not Emerge ‘Naturally’ in Multi-Agent Dialog (2017, background): “, 2014) and is witnessing a recent resurgence (Wang et al., 2016; Foerster et al., 2016; Sukhbaatar et al., 2016; Jorge et al., 2016; Lazaridou et al., 2017; Havrylov and Titov, 2017; Mordatch and Abbeel, 2017; Das et al., 2017b) – is situated language learning.”
Adversarial Language Games for Advanced Natural Language Intelligence (2019, background): “…games, where one agent needs to select a spe-ciﬁc object from candidates based on the descriptions from another agent (Lazaridou et al., 2017; Havrylov and Titov, 2017; Bouchacourt and Ba-roni, 2018; Kharitonov et al., 2019), or two agents with incomplete private information communicate to…”
Emergence of Linguistic Communication from Referential Games with Symbolic and Pixel Input (2018, result): “This contrasts with the cases of Havrylov & Titov (2017) and Lazaridou et al. (2017) who use pre-trained visual vectors, and qualitatively observe that the induced communication protocols encode information about objects.”
Measuring Compositionality in Representation Learning (2019, background): “Since then, numerous other approaches for compositional representation learning have been proposed, with (Mitchell & Lapata, 2008; Irsoy & Cardie, 2014) and without (Dircks & Stoness, 1999; Havrylov & Titov, 2017) the scaffolding of explicit composition operations built into the model.”
Learning Decision Trees Recurrently Through Communication (2021, methodology): “Learning to communicate in a multi-agent setting has gained interest with the emergence of deep reinforcement learning [15, 20, 37, 5, 28, 12, 9, 38, 36].”
Prosocial learning agents solve generalized Stag Hunts better than selfish ones (2017, background): “…can interact with others (Silver et al. 2016; Lazaridou, Peysakhovich, and Baroni 2017; Das et al. 2017; Lowe et al. 2017; Foer-ster et al. 2017; Evtimova et al. 2017; Foerster et al. 2016; Havrylov and Titov 2017; Lewis et al. 2017; Heinrich and Silver 2016; Peng et al. 2017; Leibo et al. 2017).”
Interactive Learning from Activity Description (2021, methodology): “Finally, our work also fundamentally differs from work on (RL-based) emergent language (Foerster et al., 2016; Lazaridou et al., 2017; Havrylov & Titov, 2017; Das et al., 2017; Evtimova et al., 2018; Kottur et al., 2017) in that we assume the teacher speaks a ﬁxed, well-formed language, whereas in…”
Multi-agent deep reinforcement learning: a survey (2021, background): “Many studies are concerned with the development of communication in referential games grounded on visual perception as it can be found in Choi et al. (2018), Evtimova et al. (2018), Havrylov and Titov (2017), Jorge et al. (2016), Lazaridou et al. (2018) and Lee et al. (2017).”
Iterated learning for emergent systematicity in VQA (2021, methodology): “Most approaches to learning a discrete communication protocol between agents use reinforcement learning (Foerster et al., 2016; Lazaridou et al., 2017; Kottur et al., 2017; Jorge et al., 2016; Havrylov & Titov, 2017).”
Transparency: Motivations and Challenges (2019, background): “Exciting work has begun to explore this direction, looking for ways to enable multiple agents to cooperate effectively [20, 28, 47].”