Learning to Play Guess Who? and Inventing a Grounded Language as a Consequence

By Emilio Jorge, Mikael Kågebäck, Emil Gustavsson. 2016

70 citations recorded by the source corpus.

Learning your first language is an incredible feat and not easily duplicated. Doing this using nothing but a few pictureless books, a corpus, would likely be impossible even for humans. As an alternative we propose to use situated interactions between agents as a driving force for communication, and the framework of Deep Recurrent Q-Networks (DRQN) for learning a common language grounded in the provided environment. We task the agents with interactive image search in the form of the game Guess Who?. The images from the game provide a non trivial environment for the agents to discuss and a natural grounding for the concepts they decide to encode in their communication. Our experiments show that it is possible to learn this task using DRQN and even more importantly that the words the agents use correspond to physical attributes present in the images that make up the agents environment.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Counterfactual Multi-Agent Policy Gradients (2017, background): “…over R t , V π ( s t ) = E s t +1: ∞ , u t : ∞ [ R t | s t ] , and an action-value function Following previous work (Oliehoek et al., 2008; Kraemer & Banerjee, 2016; Foerster et al., 2016; Jorge et al., 2016), our problem setting allows centralised training but requires decentralised execution.”
Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning (2017, background): “Foerster et al. (2016) pro-pose DIAL, a end-to-end differentiable architecture that allows agents to learn to communicate and has since used by Jorge et al. (2016) in a similar setting.”
QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2018, background): “The paradigm of centralised training with decentralised execution (Oliehoek et al., 2008; Kraemer & Banerjee, 2016) has recently attracted attention in the RL community (Jorge et al., 2016; Foerster et al., 2018).”
Multi-Agent Common Knowledge Reinforcement Learning (2018, methodology): “Following previous work (Rashid et al., 2018a; Foerster et al., 2016; 2017; 2018; Kraemer & Baner-jee, 2016; Jorge et al., 2016), we allow centralised learning of decentralised policies.”
Emergent Language in a Multi-Modal, Multi-Step Referential Game (2017, methodology): “Unlike Jorge et al. [14], we let S be shared between the two agents, which makes the proposed game a more realistic proxy to natural language conversations where two parties share a single vocabulary.”
Iterated learning for emergent systematicity in VQA (2021, background): “Most approaches to learning a discrete communication protocol between agents use reinforcement learning (Foerster et al., 2016; Lazaridou et al., 2017; Kottur et al., 2017; Jorge et al., 2016; Havrylov & Titov, 2017).”
Maintaining cooperation in complex social dilemmas using deep reinforcement learning (2017, background): “, 2017; Peysakhovich & Lerer, 2017) of which language emergence games are an important special case (Lazaridou et al., 2017; Das et al., 2017; Evtimova et al., 2017; Havrylov & Titov, 2017; Jorge et al., 2016).”
When Does Communication Learning Need Hierarchical Multi-Agent Deep Reinforcement Learning (2019, methodology): “DIAL was the model for Jorge et al. ’ s successful communication sequence of dialog remembered by both agents to solve the Guess Who game (Jorge et al. 2016).”
Consequentialist conditional cooperation in social dilemmas with imperfect information (2017, background): “…2017), and the emergence of language (Lazaridou, Peysakhovich, and Baroni 2017; Das et al. 2017; Evtimova et al. 2017; Havrylov and Titov 2017; Jorge, Kågebäck, and Gustavsson 2016). lying environment is only partially observed.2 The simplest social dilemma is the Prisoner’s Dilemma (PD) in…”
TarMAC: Targeted Multi-Agent Communication (2018, background): “While (Foerster et al., 2016; Jorge et al., 2016; Das et al., 2017; Kottur et al., 2017; Mordatch & Abbeel, 2017; Lazaridou et al., 2017) constrain agents to communicate with discrete symbols with the explicit goal to study emergence of language, our work operates in the paradigm of learning a…”
A review of cooperative multi-agent deep reinforcement learning (2019, background): “Learn to Communicate To improve the communication efficiency, we allow the agents to learn what to send, when to send, and send that to which agents [56, 88, 141]”
Natural Language Does Not Emerge ‘Naturally’ in Multi-Agent Dialog (2017, background): “, 2014) and is witnessing a recent resurgence (Wang et al., 2016; Foerster et al., 2016; Sukhbaatar et al., 2016; Jorge et al., 2016; Lazaridou et al., 2017; Havrylov and Titov, 2017; Mordatch and Abbeel, 2017; Das et al., 2017b) – is situated language learning.”
Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence (2019, unknown): “Other ideas include using joint action-value function [17, 26, 49, 54] and value function factorization [27, 44, 77], addressing the variance problem by a large batch size [5], learning grounded cooperative communication protocols between agents [18, 22, 30, 40, 50, 62, 89].”
Emergent Translation in Multi-Agent Communication (2017, background): “This work has been extended to induce communication protocols that are more similar to human language, allowing multi-turn communication (Jorge et al., 2016), adaptive communication bandwidth (Havrylov & Titov, 2017) and multi-turn communication with a variable-length conversation (Evtimova et al.”
Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols (2017, methodology): “Neural network models have been shown to be able to successfully induce a communication protocol for this setting (Lazaridou et al., 2016; Jorge et al., 2016; Foerster et al., 2016; Sukhbaatar et al., 2016).”
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms (2019, background): “Following [21], various works [343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354] have proposed a variety of neural network architectures to foster communication among agents.”