SemEval-2018 Task 1: Affect in Tweets

By Saif M. Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, Svetlana Kiritchenko. 2018

940 citations recorded by the source corpus.

We present the SemEval-2018 Task 1: Affect in Tweets, which includes an array of subtasks on inferring the affectual state of a person from their tweet. For each task, we created labeled data from English, Arabic, and Spanish tweets. The individual tasks are: 1. emotion intensity regression, 2. emotion intensity ordinal classification, 3. valence (sentiment) regression, 4. valence ordinal classification, and 5. emotion classification. Seventy-five teams (about 200 team members) participated in the shared task. We summarize the methods, resources, and tools used by the participating teams, with a focus on the techniques and resources that are particularly useful. We also analyze systems for consistent bias towards a particular race or gender. The data is made freely available to further improve our understanding of how people convey emotions through language.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Experimental and Cognitive Psychology

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor (2019, background): “One is that they get easily propagated both in science itself (Jha and Mamidi 2017; Gebru et al. 2018; Mohammad et al. 2018; Hall Maudslay et al. 2019), also outside NLP and AI (McQuillan 2018) and”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, background): “– TweetEval: Emotion Recognition (Mohammad et al., 2018), Emoji Prediction (Barbieri et al., 2018), Irony Detection (Van Hee et al., 2018), Hate Speech Detection (Basile et al., 2019), Offensive Language Identification (Zampieri et al., 2019), Sentiment Analysis (Rosenthal et al., 2017), Stance…”
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021, background): “…me really excited for the next 3 weeks of my life" NO IRONY: “@user on stage at #flzjingleball at the @user in #Tampa #iheartradio" SemEval 2018 Task 1 (Barbieri et al., 2020; Mohammad et al., 2018) This task classifies a tweet as one of 4 emotion types {“sadness", “joy", “anger", “optimism"}.”
Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems (2018, methodology): “We used the EEC as a supplementary test set in a recent shared task on predicting sentiment and emotion intensity in tweets, SemEval-2018 Task 1: Affect in Tweets (Mohammad et al., 2018).”
Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset (2018, background): “…research in distributional representation approaches for many emotion classiﬁcation tasks (Duppada et al., 2018; Park et al., 2018; Xu et al., 2018; Mohammad et al., 2018) that build on deep networks pretrained on large-scale weakly-labelled data such as emo-jis (Felbo et al., 2017) or hashtags…”
TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification (2020, methodology): “We also include an SVM-based baseline with both word and character n-gram features, a model and feature set that has seen great success in recent Twitter-based shared tasks such as emoji prediction (C¸¨oltekin and Rama, 2018) and stance prediction (Mohammad et al., 2018).”
Obtaining Reliable Human Ratings of Valence, Arousal, and Dominance for 20,000 English Words (2018, neutral): “…similarity (Jurgens et al., 2012), word-sense disambiguation (Jurgens, 2013), word–sentiment intensity (Kiritchenko and Mohammad, 2016), word–emotion intensity (Mohammad, 2018), and tweet–emotion intensity (Mohammad and Bravo-Marquez, 2017; Mohammad et al., 2018; Mohammad and Kiritchenko, 2018).”
Sentiment Analysis: Detecting Valence, Emotions, and Other Affectual States from Text (2016, methodology): “The SemEval-2018 Task 1: Affect in Tweets, is particularly notable, because it includes an array of subtasks on inferring both emotion classeses and emotion intensity, provides labeled data for English, Arabic, and Spanish tweets, and for the first time in an NLP shared task, analyzed systems for bi”
Learning and Evaluating Emotion Lexicons for 91 Languages (2020, background): “…sentence-level emotion models so that the performance of even the most sophisticated neural network gets boosted (Mohammad and Bravo-Marquez, 2017; Mohammad et al., 2018; De Bruyne et al., 2019), or rely on them in a keyword-spotting approach when no training data is available, e.g., for studies…”
Good Secretaries, Bad Truck Drivers? Occupational Gender Stereotypes in Sentiment Analysis (2019, methodology): “It was initially used to evaluate systems that predicted emotion and valence of Tweets (Mohammad et al., 2018).”
VAuLT: Augmenting the Vision-and-Language Transformer with the Propagation of Deep Language Representations (2022, background): “Researchers tend to focus on single modalities when analyzing emotion and sentiment in social media posts, which in the majority of cases is text (Mohammad et al. 2018; Sailu-naz and Alhajj 2019; Liew, Turtle, and Liddy 2016; Dem-szky et al. 2020; Abdul-Mageed and Ungar 2017).”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, background): “Visual Dialogue; Speaker Consistency and Personality Response Shuster et al. (2018)”
Emotionally-Aware Chatbots: A Survey (2019, methodology): “Some tasks were focusing on classifying utterance into several categories of emotion [34].”
Joint Modelling of Emotion and Abusive Language Detection (2020, methodology): “Emotion (SemEval18) This dataset is from SemEval-2018 Task 1: Affect in Tweets (Mohammad et al., 2018), and specifically from Subtask 5 which is a multilabel classification of 11 emotion labels that best represent the mental state of the author of a tweet.”
XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond (2021, neutral): “TweetEval is composed of seven tasks: emoji prediction (Barbieri et al., 2018), emotion recognition (Mohammad et al., 2018), hate speech detection (Basile et al., 2019), irony detection (Van Hee et al., 2018), of-fensive language identiﬁcation (Zampieri et al., 2019), sentiment analysis (Rosenthal…”
Meta-tuning Language Models to Answer Prompts Better (2021, background): “…me really excited for the next 3 weeks of my life " N O I RONY : “ @user on stage at #ﬂzjingleball at the @user in #Tampa #iheartradio " SemEval 2018 Task 1 (Barbieri et al., 2020; Mohammad et al., 2018) This task classiﬁes a tweet as one of 4 emotion types {“sadness", “joy", “anger", “optimism"}.”
Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics (2021, methodology): “We ex-periment on synthetic data created using hand-crafted templates, as is common in the literature (Dixon et al., 2018; Kiritchenko and Mohammad, 2018; Kurita et al., 2019; Huang et al., 2020a).”