Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning

By Angeliki Lazaridou, Anna Potapenko, Olivier Tieleman. 2020

114 citations recorded by the source corpus.

We present a method for combining multi-agent communication and traditional data-driven approaches to natural language learning, with an end goal of teaching agents to communicate with humans in natural language. Our starting point is a language model that has been trained on generic, not task-specific language data. We then place this model in a multi-agent self-play environment that generates task-specific rewards used to adapt or modulate the model, turning it into a task-conditional language model. We introduce a new way for combining the two types of learning based on the idea of reranking language model samples, and show that this method outperforms others in communicating with humans in a visual referential communication task. Finally, we present a taxonomy of different types of language drift that can occur alongside a set of measures to detect them.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, LLM Agents & Tool Use

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In LLM Agents & Tool Use

Citations in the corpus

WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, methodology): “Prior work suggests that directly fine-tuning text generation via RL might lead to language drifting [24] and deteriorated performance.”
Keep CALM and Explore: Language Models for Action Generation in Text-based Games (2020, background): “More broadly, the concept of generating candidates and re-ranking has been studied in other interactive lanugage tasks such as dialogue (Zhao and Eskenazi, 2016; Williams et al., 2017; Song et al., 2016; Chen et al., 2017) and communication games (Lazaridou et al., 2020).”
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “However, there are limits to the types of Theory of Mind that can be learned from interactive simulations, which are often task-speciﬁc (e.g., describing objects in an image; Lazaridou et al., 2020; Steinert-Threlkeld et al., 2022).”
Quark: Controllable Text Generation with Reinforced Unlearning (2022, background): “, navigation [76, 75], multi-agent communication [34], image captioning [58, 6, 59], etc.”
FUDGE: Controlled Text Generation With Future Discriminators (2021, background): “However, it is difficult to isolate the desired attribute from the distribution shift between G and the finetuning dataset (Hu et al., 2017; John et al., 2018; Lazaridou et al., 2020), i.e., it is nontrivial to preserve the desirable qualities of the P (X) modeled by G.”
Learning Decision Trees Recurrently Through Communication (2021, background): “Learning to communicate in a multi-agent setting has gained interest with the emergence of deep reinforcement learning [15, 20, 37, 5, 28, 12, 9, 38, 36].”
Emergent Communication of Generalizations (2021, methodology): “A finer measure of compositionality often used in the literature (Lazaridou et al., 2018; Li and Bowling, 2019; Lazaridou et al., 2020) is topographic ⇢ (Brighton and Kirby, 2006).”
A General Theoretical Paradigm to Understand Learning from Human Preferences (2023, background): “The constraint of staying close to a known reference policy can be satisfied e.g., by using KL regularisation (Geist et al., 2019) and its role is to avoid model drift (Lazaridou et al., 2020; Lu et al., 2020).”
Intra-agent speech permits zero-shot task acquisition (2022, background): “VAEs with language latent variables have also been studied in language summarization [47], and in the context of multi-agent communication with natural language [48, 49].”
Learning Natural Language Generation with Truncated Reinforcement Learning (2022, background): “Besides, combining pre-training and fine-tuning phases either barely change the policy distribution, or induces language drift (Lazaridou et al., 2020; Lu et al., 2020b), i.e the generated language drifts semantically or syntactically from natural language.”
Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language? (2020, background): “In concurrent work, Lazaridou et al. (2020) learn NL protocols for an image-based reference game by pretraining with image captions.”
Multitasking Inhibits Semantic Drift (2021, background): “Most re-latedly to our work, Lazaridou et al. (2020) train speaker-listener agents jointly in a visual referential communication task and introduce auxiliary loss functions for stabilizing training.”