Are All Languages Equally Hard to Language-Model?

By Ryan Cotterell, Sabrina J. Mielke, Jason Eisner, Brian Roark. 2018

116 citations recorded by the source corpus.

For general modeling methods applied to diverse languages, a natural question is: how well should we expect our models to work on languages with differing typological profiles? In this work, we develop an evaluation framework for fair cross-linguistic comparison of language models, using translated text so that all models are asked to predict approximately the same information. We then conduct a study on 21 languages, demonstrating that in some languages, the textual expression of the information is harder to predict with both n-gram and LSTM language models. We show complex inflectional morphology to be a cause of performance differences among languages.

Subfields: Artificial Intelligence, Natural Language Processing, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Scaling & Learning Theory

Citations in the corpus

Neural Generation for Czech: Data and Baselines (2019, methodology): “We choose Czech as an example of a morphologically complex language (Cotterell et al., 2018) with a large set of NLP tools readily available (e.g. Popel and ˇZabokrtsk´y, 2010; Strakov´a et al., 2014; Straka and Strakov´a, 2017).”
Troubling Trends in Machine Learning Scholarship (2018, background): “Understanding can also come from robustness checks (as in [15], which discovers that existing language models handle inflectional morphology poorly) as well as qualitative error analysis [40].”
Processing South Asian Languages Written in the Latin Script: the Dakshina Dataset (2020, background): “Additionally, since within each language we have parallel native script and romanized corpora that we are training and evaluating on, we can follow Cotterell et al. (2018) and Mielke et al. (2019) in comparing language modeling results across parallel data samples by normalizing with a common…”
mGPT: Few-Shot Learners Go Multilingual (2022, methodology): “We inferred the perplexity of an input text as follows (Cotterell et al., 2018; Lau et al., 2020):”
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, methodology): “3) in languages other than English, in particular those with more productive morphology and thus higher type-token-ratio, removing rare words is infeasible (Cotterell et al., 2018; Mielke et al., 2019).”
Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages (2019, methodology): “In another recent study, Cotterell et al. (2018) measured the ability of RNN and n-gram models to perform character-level language modeling in a sample of languages, using a parallel corpus; the main typological property of interest in that study was morphological complexity.”
Choosing Transfer Languages for Cross-Lingual Learning (2019, background): “The relationship between linguistic typology and statistical modeling has been studied by Gerz et al. (2018) and Cotterell et al. (2018), with a focus on language modeling.”
Incorporating Word and Subword Units in Unsupervised Machine Translation Using Language Model Rescoring (2019, unknown): “Language Model According to the ﬁndings in Cotterell et al. (2018), the morphological richness of a language is closely related to the performance of the model, which indicates that the language models will be extremely hard to train for Czech, as it is one of the most complex languages.”