langid.py: An Off-the-shelf Language Identification Tool

By Marco Lui, Timothy Baldwin. 2012

690 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Learning Word Vectors for 157 Languages (2018, positive): “Our approach compares favorably to existing methods such as langid.py (Lui and Baldwin, 2012), while being much faster.”
Demographic Dialectal Variation in Social Media: A Case Study of African-American English (2016, neutral): “Lui and Baldwin (2012) develop langid.”
OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles (2016, methodology): “• Probability that the specified language of the subtitle matches the actual language used in the subtitle text, based on the output of the langid language identification tool (Lui and Baldwin, 2012).”
Incorporating Dialectal Variability for Socially Equitable Language Identification (2017, methodology): “Multiple systems have been proposed for broad-coverage LID at the global level (McCandless, 2010; Lui and Baldwin, 2012; Brown, 2014; Jaech et al., 2016).”
Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American English (2017, background): “…which are popular and straightforward for engineers to use when building applications: • langid.py (software): Oneofthemostpopular opensource languageidentificationtools, langid.py wasoriginallytrained on over 97 languages and evaluated on both traditional corpora and Twitter messages [16].”
Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations (2019, methodology): “The latent language identity z = φ ( y j ) is estimated by an external language identiﬁcation tool given the actual output (Lui and Baldwin, 2012).”
Results of the WNUT2017 Shared Task on Novel and Emerging Entity Recognition (2017, positive): “Non-English comments were removed with langid.py (Lui and Baldwin, 2012).”
Analyzing Biases in Human Perception of User Age and Gender from Text (2016, methodology): “These are ﬁltered for English language using an automatic method (Lui and Baldwin, 2012) and duplicate tweets are eliminated (i.e., having the same ﬁrst 6 tokens) as these are usually generated automatically by apps.”
CUNI Submission for Low-Resource Languages in WMT News 2019 (2019, methodology): “Therefore, we used language identiﬁcation tool by Lui and Baldwin (2012) on the Common Crawl corpus and dropped all sentences automatically annotated as a different language than Gujarati or Kazakh respectively.”
Can You Translate that into Man? Commercial Machine Translation Systems Include Stylistic Biases (2020, methodology): “For each language, we restrict ourselves to reviews written in the respective language (according to langid 1 (Lui and Baldwin, 2012)) that have both age and gender information.”
Gender Profiling for Slovene Twitter communication: the Influence of Gender Marking, Content and Style (2017, methodology): “For this the langid.py program (Lui and Baldwin, 2012) was used.”
Dirt Cheap Web-Scale Parallel Text from the Common Crawl (2013, methodology): “We used the “langid.py” tool (Lui and Baldwin, 2012) at the segment level, and report the percentage of sentence pairs where both sentences were recognized as the correct language.”