Code Mixing: A Challenge for Language Identification in the Language of Social Media
By Utsab Barman, Amitava Das, Joachim Wagner, Jennifer Foster. CodeSwitch@EMNLP, 2014
312 citations recorded by the source corpus.
In social media communication, multilingual speakers often switch between languages, and, in such an environment, automatic language identification becomes both a necessary and challenging task. In this paper, we describe our work in progress on the problem of automatic language identification for the language of social media. We describe a new dataset that we are in the process of creating, which contains Facebook posts and comments that exhibit code mixing between Bengali, English and Hindi. We also present some preliminary word-level language identification experiments using this dataset. Different techniques are employed, including a simple unsupervised dictionary-based approach, supervised word-level classification with and without contextual clues, and sequence labelling using Conditional Random Fields. We find that the dictionary-based approach is surpassed by supervised classification and sequence labelling, and that it is important to take contextual clues into consideration.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/004c857f76779260664a9907645938c96cb1896a
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Francesco Barbieri
- José Camacho-Collados
- Leonardo Neves
- Jason Baldridge
- Keoni Mahelona
- Luis Espinosa-Anke
- Peter J. Keegan
- David Weiss
Citations in the corpus
- Language Models for Code-switch Detection of te reo Māori and English in a Low-resource Setting (2022, methodology): “The language detection task is a crucial first step for detecting code-switching (Rijhwani et al., 2017; Barman et al., 2014).”
- Corpus Creation for Sentiment Analysis in Code-Mixed Tamil-English Text (2020, background): “But social media in multilingual communities contains more code-mixed text (Barman et al., 2014; Chanda et al., 2016; Pratapa et al., 2018a; Winata et al., 2019a).”
- TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification (2020, background): “There are also other scenarios to be addressed as well, like sequence tagging (Baldwin et al., 2015; Gimpel et al., 2018), multimodality (Schifanella et al., 2016; Lu et al., 2018), and code-switching tasks (Barman et al., 2014; Vilares et al., 2016).”
- A Fast, Compact, Accurate Model for Language Identification of Codemixed Text (2018, methodology): “Then we choose uniformly between generating an intra-mix or inter-mix example, which are two of the most prominent types of codemix-ing in the real world (Barman et al., 2014; Das and Gamb¨ack, 2014).”