Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm

By Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, Sune Lehmann. 2017

813 citations recorded by the source corpus.

NLP tasks are often limited by scarcity of manually annotated data. In social media sentiment analysis and related tasks, researchers have therefore used binarized emoticons and specific hashtags as forms of distant supervision. Our paper shows that by extending the distant supervision to a more diverse set of noisy labels, the models can learn richer representations. Through emoji prediction on a dataset of 1246 million tweets containing one of 64 common emojis we obtain state-of-the-art performance on 8 benchmark datasets within emotion, sentiment and sarcasm detection using a single pretrained model. Our analyses confirm that the diversity of our emotional labels yield a performance improvement over previous distant supervision approaches.

Subfields: Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Universal Language Model Fine-tuning for Text Classification (2018, background): “This is similar to ‘ chain-thaw ’ (Felbo et al., 2017), except that we add a layer at a time to the set of ‘thawed’ layers, rather than only training a single layer at a time.”
Transfer Learning in Natural Language Processing (2019, background): “Optimization schedules for the adaptation phase can involve fine-tuning a varying portion of the pre-trained model (Long et al., 2015; Felbo et al., 2017; Howard and Ruder, 2018) with specifically designed regularization (Wiese et al.”
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks (2019, background): “While pretraining tasks have been designed with particular downstream tasks in mind (Felbo et al., 2017), we focus on pretraining tasks that seek to ar X iv :1 90 3.”
Fine-tuned Language Models for Text Classification (2018, methodology): “ning Fine-tuning, which ﬁne-tunes a pretrained model on the target task, has been used successfully for certain tasks such as sentiment analysis with distant supervision (Severyn and Moschitti ,2015; Felbo et al. 2017). In the following, we will show that ﬁne-tuning is superior to both hypercolumns ”
Knowledge Enhanced Contextual Word Representations (2019, neutral): “It is similar in spirit to the “chain-thaw” approach in Felbo et al. (2017), and is summarized in Alg.”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, methodology): “The Bot Sentiment reward is the DeepMoji sentiment computed on the bot response, User Sentiment reward is the value computed on the user response, and the Sentiment Coherence reward is based on the similarly of user and bot sentiments.”
Refining Language Models with Compositional Explanations (2021, background): “, updating layer weights [8], creating learning rate schedule [18], adding regularization [48, 28, 27], and model distillation [17, 37].”
Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems (2018, methodology): “…AIT2018 Distant Supervision Corpus that has tweets with emotion-related query terms), sentiment-labeled tweet corpora (e.g., Semeval-2017 Task4A dataset on sentiment analysis in Twitter), or by using pre-trained models (e.g., DeepMoji (Felbo et al., 2017), Skip thoughts (Kiros et al., 2015)).”
Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset (2018, background): “…tasks (Duppada et al., 2018; Park et al., 2018; Xu et al., 2018; Mohammad et al., 2018) that build on deep networks pretrained on large-scale weakly-labelled data such as emo-jis (Felbo et al., 2017) or hashtags (Mohammad, 2012), gathered from public social media content published on Twitter.”
Reducing Gender Bias in Abusive Language Detection (2018, methodology): “RNN is trained (Park and Fung, 2017), Gated Recurrent Unit (GRU) (Cho et al., 2014), and Bidirectional GRU with self-attention (α-GRU) (Pavlopoulos et al., 2017), but with a simpler mechanism used in Felbo et al. (2017). Hyperparameters are found using the validation set by ﬁnding the best performin”
Adversarial Removal of Demographic Attributes from Text Data (2018, methodology): “Here, we demonstrate leakage using a popular encoder trained for emotion detection: the Deep-Moji encoder (Felbo et al., 2017) emoji usage is highly correlated with these properties.”
Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems (2019, methodology): “Sentiment metrics To approximate emotional tone of an utterance, we use a state-of-the-art sentiment detector trained on a large Twitter corpus [32].”
Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection (2020, methodology): “Our classiﬁer is based on the DeepMoji encoder (Felbo et al., 2017), followed by a 1-hideen-layer MLP.”
SemEval-2018 Task 1: Affect in Tweets (2018, methodology): “The following models or resources were used for feature extraction: • DeepMoji (Felbo et al., 2017): a neural network for predicting emoji for tweets trained from a very large distant supervision corpus.”
Argumentative Relation Classification as Plausibility Ranking (2019, methodology): “Finally, we compute a weighted average of the final sequence of hidden states to construct a vectorized reading representation v (Felbo et al., 2017):”
CARER: Contextualized Affect Representations for Emotion Recognition (2018, methodology): “, 2017); char refers to character-level features; ngram employ unigrams, bigrams, and trigrams as features; CNNBASIC uses the proposed CNN architecture with basic patterns; EmoNet (Abdul-Mageed and Ungar, 2017) and DeepMoji (Felbo et al., 2017) are state-of-the-art emotion recognition models.”
Mining Discourse Markers for Unsupervised Sentence Representation Learning (2019, methodology): “%) using FastText langid from Grave et al. (2018), have balanced parentheses and quotes, and are mostly lowercase.”
An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models (2019, neutral): “, 2018) (Felbo et al., 2017)”
Towards Explainable NLP: A Generative Explanation Framework for Text Classification (2018, methodology): “The classiﬁer is a skip-connected model of bidirectional GRU-RNN layers (Felbo et al., 2017).”