Polyglot Neural Language Models: A Case Study in Cross-Lingual Phonetic Representation Learning

By Yulia Tsvetkov, Sunayana Sitaram, Manaal Faruqui, Guillaume Lample, Patrick Littell, David R. Mortensen, Alan W. Black, Lori S. Levin, Chris Dyer. 2016

67 citations recorded by the source corpus.

We introduce polyglot language models, recurrent neural network models trained to predict symbol sequences in many different languages using shared representations of symbols and conditioning on typological information about the language to be predicted. We apply these to the problem of modeling phone sequences---a domain in which universal symbol inventories and cross-linguistically shared feature representations are a natural fit. Intrinsic evaluation on held-out perplexity, qualitative analysis of the learned representations, and extrinsic evaluation in two downstream applications that make use of phonetic features show (i) that polyglot models better generalize to held-out data than comparable monolingual models and (ii) that polyglot phonetic feature representations are of higher quality than those learned monolingually.

Subfields: Artificial Intelligence, Large Language Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Representation & Self-Supervised Learning

Citations in the corpus

Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation (2016, background): “Multilingual language processing has also shown to be successful in domains other than translation (Gillick et al., 2016; Tsvetkov et al., 2016).”
Fully Character-Level Neural Machine Translation without Explicit Segmentation (2016, background): “Also, we drew in-spiration for our multilingual models from previous work that showed the possibility of training a single recurrent model for multiple languages in domains other than translation (Tsvetkov et al., 2016; Gillick et al., 2015).”
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2021, background): “…uses only a few examples from the low-resource domain to adapt models trained in another domain, and polyglot learning (Cotterell and Heigold, 2017; Tsvetkov et al., 2016; Mulcaire et al., 2019; Lample and Conneau, 2019) which combines resource-rich and resource-poor learning us-ing an universal…”
Phonological Features for 0-shot Multilingual Speech Synthesis (2020, background): “Producing accent-free, 0-shot speech, perhaps relying on more sophisticated representations of typology and language [11, 28] or with adversarial losses [1] remains a challenging task for future research.”
Polyglot Contextual Representations Improve Crosslingual Transfer (2019, background): “…true to some degree in the context of syntactic dependency parsing (Ammar et al., 2016), semantic role labeling (Mulcaire et al., 2018), named entity recognition (Xie et al., 2018), and language modeling for phonetic sequences (Tsvetkov et al., 2016) and for speech recognition (Ragni et al., 2016).”
One-Size-Fits-All Multilingual Models (2020, background): “A simple approach to multilingual NLP is to append a token to each input sequence identifying the language of the sample; this has proven effective for both g2p (Peters et al., 2017) and morphological inﬂection (Peters et al., 2019), and is similar to techniques for multi-lingual neural machine tran”
URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors (2017, methodology): “The recent success of “polyglot” models (Hermann and Blunsom, 2014; Faruqui and Dyer, 2014; Ammar et al., 2016; Tsvetkov et al., 2016; Daiber et al., 2016), in which a language model is trained on multiple languages and shares representations across languages, represents a promising avenue for NLP,…”