A Fast, Compact, Accurate Model for Language Identification of Codemixed Text

By Jason Riesa, D. Gillick, Anton Bakalov, Jason Baldridge, David Weiss. 2018

36 citations recorded by the source corpus.

We address fine-grained multilingual language identification: providing a language code for every token in a sentence, including codemixed text containing multiple languages. Such text is prevalent online, in documents, social media, and message boards. We show that a feed-forward network with a simple globally constrained decoder can accurately and rapidly label both codemixed and monolingual text in 100 languages and 100 language pairs. This model outperforms previously published multilingual approaches in terms of both accuracy and speed, yielding an 800x speed-up and a 19.5% averaged absolute gain on three codemixed datasets. It furthermore outperforms several benchmark systems on monolingual language identification.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Federated Learning for Emoji Prediction in a Mobile Keyboard (2019, background): “The logs are filtered further to only include sentences that are labeled as English with high confidence by a language detection model (Botha et al., 2017; Zhang et al., 2018).”
Entity-Switched Datasets: An Approach to Auditing the In-Domain Robustness of Named Entity Recognition Models (2020, background): “Language identiﬁcation is also highly accurate on standard datasets (Zhang et al., 2018) but may fail to recognize di-alects, e.g., failing to identifying African American English as English (Blodgett et al., 2016).”
Dialect-robust Evaluation of Generated Text (2022, methodology): “We split the corpus into sentences and use a Language Identification (LangID) model (Zhang et al., 2018) by Botha et al. (2017) to identify the language and locale information for the sentences.7 Besides LangID output, mC4 provides the URL where a sentence originated from which we extract the…”
Processing South Asian Languages Written in the Latin Script: the Dakshina Dataset (2020, background): “Zhang et al. (2018) looked at large-scale language identification that included some evaluation of Hindi written in the Latin script.”