Emerging Language Spaces Learned From Massively Multilingual Corpora

By Jörg Tiedemann. 2018

25 citations recorded by the source corpus.

Translations capture important information about languages that can be used as implicit supervision in learning linguistic properties and semantic representations. In an information-centric view, translated texts may be considered as semantic mirrors of the original text and the significant variations that we can observe across various languages can be used to disambiguate a given expression using the linguistic signal that is grounded in translation. Parallel corpora consisting of massive amounts of human translations with a large linguistic variation can be applied to increase abstractions and we propose the use of highly multilingual machine translation models to find language-independent meaning representations. Our initial experiments show that neural machine translation models can indeed learn in such a setup and we can show that the learning algorithm picks up information about the relation between languages in order to optimize transfer leaning with shared parameters. The model creates a continuous language space that represents relationships in terms of geometric distances, which we can visualize to illustrate how languages cluster according to language families and groups. Does this open the door for new ideas of data-driven language typology with promising models and techniques in empirical cross-linguistic research?

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Massively Multilingual Neural Machine Translation (2019, background): “Tiedemann (2018) trained an English-centric many-to-many model on translations of the bible including 927 languages.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “( vi ) The Bible corpus is perhaps the most multilingual corpus publicly available, containing 30k sentences in over 900 languages (Tiedemann, 2018).”
Complete Multilingual Neural Machine Translation (2020, neutral): “The amount of available training data can differ a lot across language pairs and the majority of available MT training data is English-centric (Tiedemann, 2018; Arivazhagan et al., 2019b) which in”
The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation (2021, background): “, 2020) or religious texts (Christodouloupoulos and Steedman, 2015; Malaviya et al., 2017; Tiedemann, 2018; Agić and Vulić, 2019); or have low quality because they are built using automatic approaches (Zhang et al.”
Investigating Multilingual NMT Representations at Scale (2019, methodology): “Some recent works compare learned language vectors (Östling and Tiedemann, 2016; Tan et al., 2019; Tiedemann, 2018) and find conclusions similar to ours.”
Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference (2021, background): “Multilingual models learn joint representations across languages to the extent of the parameters being shared (Wu and Dredze, 2019; Tiedemann, 2018; Tan et al., 2019; Zhang et al., 2020; Östling and Tiedemann, 2016; Kudugunta et al., 2019).”