Emergent Translation in Multi-Agent Communication

By Jason Lee, Kyunghyun Cho, Jason Aaron Edward Weston, Douwe Kiela. 2017

69 citations recorded by the source corpus.

While most machine translation systems to date are trained on large parallel corpora, humans learn language in a different way: by being grounded in an environment and interacting with other humans. In this work, we propose a communication game where two agents, native speakers of their own respective languages, jointly learn to solve a visual referential task. We find that the ability to understand and translate a foreign language emerges as a means to achieve shared goals. The emergent translation is interactive and multimodal, and crucially does not require parallel corpora, but only monolingual, independent text and corresponding images. Our proposed translation model achieves this by grounding the source and target languages into a shared visual modality, and outperforms several baselines on both word-level and sentence-level translation tasks. Furthermore, we show that agents in a multilingual community learn to translate better and faster than in a bilingual communication setting.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Meta-Learning for Low-Resource Neural Machine Translation (2018, background): “For instance, Cheng et al. (2016); Chen et al. (2017); Lee et al. (2017); Chen et al. (2018) investigate the use of a pivot to build a translation path be-tween two languages even without any directed resource.”
Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering (2019, background): “This can also be seen as an instance of two agents cooperating via communication to solve a task (Lazaridou et al., 2017; Lee et al., 2018; Cao et al., 2018).”
Unsupervised Machine Translation Using Monolingual Corpora Only (2017, background): “Previous work on zero-resource machine translation has also relied on labeled information, not from the language pair of interest but from other related language pairs (Firat et al., 2016; Johnson et al., 2016; Chen et al., 2017) or from other modalities (Nakayama & Nishida, 2017; Lee et al., 2017).”
Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations (2019, background): “For instance, there have been attempts to train a single-pair model with a pivot-language (Cheng et al., 2016; Chen et al., 2017) or a pivot-image (Lee et al., 2017; Chen et al., 2018).”
Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks (2018, background): “Lee et al. (2017) demonstrated that agents speaking different languages can learn to translate in referential games.”
Multi-agent deep reinforcement learning: a survey (2021, background): “Many studies are concerned with the development of communication in referential games grounded on visual perception as it can be found in Choi et al. (2018), Evtimova et al. (2018), Havrylov and Titov (2017), Jorge et al. (2016), Lazaridou et al. (2018) and Lee et al. (2017).”
Learning Visually Grounded Sentence Representations (2017, methodology): “Similar ideas were recently successfully applied to multi-modal machine translation (Elliott and Kádár, 2017; Gella et al., 2017; Lee et al., 2017).”