CROWDSOURCING A WORD–EMOTION ASSOCIATION LEXICON

By Saif M. Mohammad, Peter D. Turney. International Conference on Climate Informatics, 2013

2,675 citations recorded by the source corpus.

Even though considerable attention has been given to the polarity of words (positive and negative) and the creation of large polarity lexicons, research in emotion analysis has had to rely on limited and small emotion lexicons. In this paper, we show how the combined strength and wisdom of the crowds can be used to generate a large, high‐quality, word–emotion and word–polarity association lexicon quickly and inexpensively. We enumerate the challenges in emotion annotation in a crowdsourcing scenario and propose solutions to address them. Most notably, in addition to questions about emotions associated with terms, we show how the inclusion of a word choice question can discourage malicious data entry, help to identify instances where the annotator may not be familiar with the target term (allowing us to reject such annotations), and help to obtain annotations at sense level (rather than at word level). We conducted experiments on how to formulate the emotion‐annotation questions, and show that asking if a term is associated with an emotion leads to markedly higher interannotator agreement than that obtained by asking if a term evokes an emotion.

Subfields: Artificial Intelligence, Information Systems, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/54227c063bb04489caffd65ff9fc6218788ddb25

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Statistical and Nonlinear Physics

Citations in the corpus

Obtaining Reliable Human Ratings of Valence, Arousal, and Dominance for 20,000 English Words (2018, methodology): “• All terms in the NRC Emotion Lexicon (Mohammad and Turney, 2013).”
Modeling Naive Psychology of Characters in Simple Commonsense Stories (2018, methodology): “Among several theories of emotion, we work with the “wheel of emotions” of Plutchik (1980), as it has been a common choice in prior literature on emotion categorization (Mohammad and Turney, 2013; Zhou et al., 2016).”
Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research (2020, background): “NRC Lexicon, Mohammad & Turney [148], [149] is another”
How Translation Alters Sentiment (2016, methodology): “The sentiment lexicon features are derived from existing, general-purpose, manual lexicons, namely NRC Emotion Lexicon (Mohammad & Turney, 2010, 2013), Bing Liu Lexicon (Hu & Liu, 2004), and MPQA Subjectivity Lexicon (Wilson et al., 2005).”
Sentiments and emotions evoked by news headlines of coronavirus disease (COVID-19) outbreak (2020, methodology): “By using the Lexicon method recommended by Saif (Mohammad and Turney, 2013) in which words are associated with emotions, sentiments and opinion categories (Wilson et al.”
Building Large-Scale Twitter-Specific Sentiment Lexicon : A Representation Learning Approach (2014, neutral): “, 2005), NRC-Emotion10 (Mohammad and Turney, 2012), HashtagLex and Sentiment140Lex11 (Mohammad et al.”
Learning and Evaluating Emotion Lexicons for 91 Languages (2020, methodology): “Making indirect use of this observation, Mohammad and Turney (2013) offer machine-translated versions of their NRC Emotion Lexicon.”
Beyond Sentiment: The Manifold of Human Emotions (2012, background): “The work described in (Strapparava and Valitutti, 2004; Quan and Ren, 2009; Mohammad and Turney, 2011) address the task of constructing a useful corpus for emotion analysis.”
A Survey on Gender Bias in Natural Language Processing (2021, background): “In particular, Nadeem et al. [2021] present StereoSet, which is a dataset to measure stereotypical biases in gender, among other domains.”