Learning Language Representations for Typology Prediction

By Chaitanya Malaviya, Graham Neubig, Patrick Littell. 2017

112 citations recorded by the source corpus.

One central mystery of neural NLP is what neural models “know” about their subject matter. When a neural machine translation system learns to translate from one language to another, does it learn the syntax or semantics of the languages? Can this knowledge be extracted from the system to fill holes in human scientific knowledge? Existing typological databases contain relatively full feature specifications for only a few hundred languages. Exploiting the existence of parallel texts in more than a thousand languages, we build a massive many-to-one NMT system from 1017 languages into English, and use this to predict information missing from typological databases. Experiments show that the proposed method is able to infer not only syntactic, but also phonological and phonetic inventory features, and improves over a baseline that has access to information about the languages geographic and phylogenetic neighbors.

Subfields: Artificial Intelligence, Cultural Studies, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cultural Studies

In Natural Language Processing

Citations in the corpus

Massively Multilingual Neural Machine Translation (2019, background): “Similarly, Malaviya et al. (2017) trained a many-to-English system including 1017 languages from bible translations, and used it to infer typological features for the different languages (without evaluating the translation quality).”
Beyond English-Centric Multilingual Machine Translation (2020, background): “, 2016) as well as investigations with hundreds of languages by training on data from the Bible (Malaviya et al., 2017; Tiedemann, 2018).”
The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation (2021, background): “, 2020) or religious texts (Christodouloupoulos and Steedman, 2015; Malaviya et al., 2017; Tiedemann, 2018; Agić and Vulić, 2019); or have low quality because they are built using automatic approaches (Zhang et al.”
BARTScore: Evaluating Generated Text as Text Generation (2021, methodology): “We use language vectors to measure the distance between two languages [46], and consider 6 distances, which are syntactic , geographic , phonological , genetic , inventory and featural distances.”
Multilingual Neural Machine Translation with Language Clustering (2019, methodology): “When training a multilingual model, it is common practice to add a tag to the input of encoder to indicate which language the model is currently processing (Johnson et al., 2017; Firat et al., 2016; Malaviya et al., 2017; Tiedemann and ¨Ostling, 2017).”
A Survey of Multilingual Neural Machine Translation (2019, background): “It has further been translated in more than 1,000 languages and used for multilingual studies [96, 111].”
UDapter: Language Adaptation for Truly Universal Dependency Parsing (2020, methodology): “(2019, figure 8) using the language vectors learned by Malaviya et al. (2017). Figure 5 illustrates 2D vector spaces generated for the typological feature vectors lt (A) and the language embeddings le learned by UDapter with or without typological features (B and C respectively).”
Balancing Training for Multilingual Neural Machine Translation (2020, background): “…n model gradients, 3 this approximation does not require extra memory even if n is large, which is important given recent efforts to scale multilingual training to 100+ (Arivazhagan et al., 2019; Aharoni et al., 2019) or even 1000+ languages ( ¨Ostling and Tiedemann, 2017; Malaviya et al., 2017).”
Polyglot Prompt: Multilingual Multitask PrompTraining (2022, background): “As a result, a ﬁne-tuned model in a data-rich language such as English can achieve decent transfer (e.g., zero-shot) performance in geographically, syntactically, or phonetically similar languages (Malaviya et al., 2017).”
Should All Cross-Lingual Embeddings Speak English? (2019, methodology): “We use data from the TED (Qi et al., 2018), OpenSubtitles (Lison and Tiedemann, 2016), WikiMa-trix (Schwenk et al., 2019), bible (Malaviya et al., 2017), and JW300 (Agi´c and Vuli´c, 2019) datasets.”