Obtaining Reliable Human Ratings of Valence, Arousal, and Dominance for 20,000 English Words

By Saif M. Mohammad. Annual Meeting of the Association for Computational Linguistics, 2018

646 citations recorded by the source corpus.

Words play a central role in language and thought. Factor analysis studies have shown that the primary dimensions of meaning are valence, arousal, and dominance (VAD). We present the NRC VAD Lexicon, which has human ratings of valence, arousal, and dominance for more than 20,000 English words. We use Best–Worst Scaling to obtain fine-grained scores and address issues of annotation consistency that plague traditional rating scale methods of annotation. We show that the ratings obtained are vastly more reliable than those in existing lexicons. We also show that there exist statistically significant differences in the shared understanding of valence, arousal, and dominance across demographic variables such as age, gender, and personality.

Subfields: Artificial Intelligence, Social Psychology, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/502c1e8bba1578241c354a54926ca913d5fa9c8a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Social Psychology

In Sociology and Political Science

Citations in the corpus

Sentiment Analysis: Detecting Valence, Emotions, and Other Affectual States from Text (2016, methodology): “…(Johnsen, Vambheim, Wynn, & Wangberg, 2014; Eichstaedt, Schwartz, Kern, Park, Labarthe, Merchant, Jha, Agrawal, Dziurzynski, Sap, et al., 2015), gender differences in how we perceive connotative word meaning (Mohammad, 2018a), and tracking well-being (Schwartz et al., 2013; Paul & Dredze, 2011).”
A Framework for the Computational Linguistic Analysis of Dehumanization (2020, methodology): “Valence scores for each word come from the word’s (or its lemma’s) entry in the NRC VAD Valence Lexicon (Mohammad, 2018), which range from 0 (most negative) to 1 (most positive).”
Transformer models for text-based emotion detection: a review of BERT-based approaches (2021, methodology): “They utilized the NRC-VAD lexicon (Mohammad 2018) to categorically map labels to the VAD dimensional space to learn VAD distributions from the input sentences categorical labels.”
Entity-Centric Contextual Affective Analysis (2019, methodology): “Furthermore, unlike the connotation frame annotations (Rashkin et al., 2016; Sap et al., 2017), the VAD lexicons used in this work were speciﬁcally motivated by social psychology literature on this topic, which inﬂuenced the annotation scheme (Mohammad, 2018).”
Natural Language Processing with Commonsense Knowledge: A Survey (2021, neutral): “…the KEMP model, which enhances empathetic dialogue generation by constructing an emotional context graph from external knowledge (ConceptNet and NRC VAD [123]) and employing an emotional cross-attention mechanism to learn and utilize emotional dependencies for generating more empathetic responses.”
On Measures of Biases and Harms in NLP (2021, background): “…in NLP typically have been categorized broadly into intrinsic or extrinsic evaluations based on whether they measure biased associations within the word embedding spaces (Caliskan et al., 2017) or biased decisions from models for specific tasks (Mohammad, 2018; Webster et al., 2019), respectively.”
GoEmotions: A Dataset of Fine-Grained Emotions (2020, background): “Feature-based models often make use of handbuilt lexicons, such as the Valence Arousal Dominance Lexicon (Mohammad, 2018).”
Learning and Evaluating Emotion Lexicons for 91 Languages (2020, methodology): “…an increasing number of researchers focus on more expressive encodings for affective states inspired by distinct lines of work in psychology (Yu et al., 2016; Buechel and Hahn, 2017; Sedoc et al., 2017; Abdul-Mageed and Ungar, 2017; Bostan and Klinger, 2018; Mohammad, 2018; Troiano et al., 2019).”
SemEval-2018 Task 1: Affect in Tweets (2018, background): “Similarly, tasks on valence, arousal, and dominance can benefit from lexicons such as ANEW (Bradley and Lang, 1999) and the newly created NRC Valence-Arousal-Dominance Lexicon (Mohammad, 2018a), which has entries for about 20,000 English terms.”
PoKi: A Large Dataset of Poems by Children (2020, methodology): “V correlations for the words in PoKi, poems by adults, and also for all the words in the NRC VAD Lexicon.”
A Survey on Gender Bias in Natural Language Processing (2021, methodology): “For datasets where each instance contains at least two versions of the same template sentence, e.g., male and female, the paired t-test has been used to measure if the mean predicted class probabilities are different across genders [Bhaskaran and Bhallamudi 2019; Kiritchenko and Mohammad 2018].”
A Comparative Study on Affective Cues in Text Embeddings Across Psychological Emotion Theories (2026, unknown): “We favored structured lexica, i.e., NRC-VAD [31,33] and NRC-EIL [32] to reduce ambiguity in syntax and better match the conditions under which the corresponding theories have been studied.”