SemEval-2017 Task 4: Sentiment Analysis in Twitter
By Sara Rosenthal, Noura Farra, Preslav Nakov. 2017
911 citations recorded by the source corpus.
This paper describes the fifth year of the Sentiment Analysis in Twitter task. SemEval-2017 Task 4 continues with a rerun of the subtasks of SemEval-2016 Task 4, which include identifying the overall sentiment of the tweet, sentiment towards a topic with classification on a two-point and on a five-point ordinal scale, and quantification of the distribution of sentiment towards a topic across a number of tweets: again on a two-point and on a five-point ordinal scale. Compared to 2016, we made two changes: (i) we introduced a new language, Arabic, for all subtasks, and (ii) we made available information from the profiles of the Twitter users who posted the target tweets. The task continues to be very popular, with a total of 48 teams participating this year.
Subfields: Artificial Intelligence, Evaluation & Benchmarks, Sentiment & Opinion Analysis
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yonatan Belinkov
- Mohit Bansal
- Jianfeng Gao
- Douwe Kiela
- Luke S. Zettlemoyer
- Noah A. Smith
- Percy Liang
- Graham Neubig
In Evaluation & Benchmarks
- Douwe Kiela
- James R. Glass
- Christian Hardmeier
- Eduard H. Hovy
- Mauro Cettolo
- Veronique Hoste
- Divyansh Kaushik
- Zachary Chase Lipton
In Sentiment & Opinion Analysis
- Christos Baziotis
- José Camacho-Collados
- Francesco Barbieri
- Alexandros Potamianos
- Alexandra Chronopoulou
- Muhammad Abdul-Mageed
- AbdelRahim Elmadany
- Eduard H. Hovy
Citations in the corpus
- Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021, neutral): “SemEval 2017 Task 4 (Barbieri et al., 2020) (Rosenthal et al., 2017).”
- Learning the Difference that Makes a Difference with Counterfactually-Augmented Data (2019, methodology): “We evaluate models on Amazon reviews (Ni et al., 2019) on data aggregated over six genres: beauty, fashion, appliances, giftcards, magazines, and software , the Twitter sentiment dataset (Rosenthal et al., 2017), 4 and Yelp reviews released as part of the Yelp dataset challenge.”
- TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification (2020, methodology): “We use the Semeval2017 dataset for Subtask A (Rosenthal et al., 2019),”
- CARER: Contextualized Affect Representations for Emotion Recognition (2018, background): “Emotion Corpora: There are several affective datasets such as SemEval-2017 Task 4 (Rosenthal et al., 2017) and Olympic games dataset (Sintsova et al., 2013).”
- An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models (2019, methodology): “We evaluate our model on five datasets: Sent17 for sentiment analysis (Rosenthal et al., 2017), PsychExp for emotion recognition (Wall-bott and Scherer, 1986), Irony18 for irony detection (Van Hee et al., 2018), SCv1 and SCv2 for sarcasm detection (Oraby et al., 2016; Lukin and Walker, 2013).”
- Predicting Factuality of Reporting and Bias of News Media Sources (2018, background): “See (Baccianella et al., 2009; Rosenthal et al., 2017) for more details about MAE M vs. MAE.”
- DynaSent: A Dynamic Benchmark for Sentiment Analysis (2020, background): “SemEval 2016 and 2017 (Nakov et al., 2016; Rosenthal et al., 2017) offered Twitter-based sentiment datasets.”
- Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision (2020, methodology): “upervision with relations between pairs of related examples. We additionally evaluate the model on out-of-sample data with three additional test sets: Amazon Reviews [45], Semeval 2017 (Twitter data) [55], and Yelp reviews [77]. The model trained on IMDb is applied without any ne-tuning to these, wh”
- ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic (2020, methodology): “Whenever no clear 2020), ArSAS (Elmadany et al., 2018), ArSenD-Lev (Baly et al., 2019), ASTD (Nabil et al., 2015), AWATIF (Abdul-Mageed and Diab, 2012), BBNS & SYTS (Salameh et al., 2015), CAMel Sent (Obeid et al., 2020), HARD (Elnagar et al., 2018), LABR (Aly and Atiya, 2013), Twitter Abdullah (Ab-”
- A review on sentiment analysis from social media platforms (2023, methodology): “In the case of sentiment analysis of twitter corpora, we found SemEval 2017 Rosenthal et al. (2017) to be a dataset frequently used as a reference and benchmark.”
- Findings of the 2017 DiscoMT Shared Task on Cross-lingual Pronoun Prediction (2017, methodology): “While in 2015 we used macro-averaged F 1 as an official evaluation measure, this year we followed the setup of 2016, where we switched to macro-averaged recall , which was also recently adopted by some other competitions, e.g., by SemEval-2016/2017 Task 4 (Nakov et al., 2016; Rosenthal et al., 2017).”
- GPT is an effective tool for multilingual psychological text analysis (2024, neutral): “Even the oldest GPT model we analyzed, GPT - 3.5 Turbo, achieved good performance at predicting human ratings in both English (Accuracy = 0.673, F1 = 0.685) and Arabic (Accuracy = 0.700, F1 = 0.720) (Table 3).”
- Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach (2018, methodology): “Sentiment analysis is also related to our work (Socher et al., 2011; Pontiki et al., 2015; Rosenthal et al., 2017; Chen et al., 2017; Ma et al., 2017, 2018b).”
- Modeling Empathy and Distress in Reaction to News Stories (2018, background): “However, the vast majority of contributions has focused on polarity prediction , typically only distinguishing between positive and negative feeling or evaluation, usually in social media postings or product reviews (Rosenthal et al., 2017; Socher et al., 2013).”
- TimeLMs: Diachronic Language Models from Twitter (2022, background): “…et al., 2018), emotion recognition (Mohammad et al., 2018), hate speech detection (Basile et al., 2019), irony detection (Van Hee et al., 2018), offensive language identification (Zampieri et al., 2019), sentiment analysis (Rosenthal et al., 2017) and stance detection (Mohammad et al., 2016).”
- Meta-tuning Language Models to Answer Prompts Better (2021, neutral): “SemEval 2017 Task 4 (Barbieri et al., 2020) (Rosenthal et al., 2017).”
- XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond (2021, neutral): “Arabic SemEval-17 (Rosenthal et al., 2017) (Benamara et al., 2017) 2014-2016 Same German SB-10K (Cieliebak et al., 2017) 8/13-10/13 Same Hindi SAIL 2015 (Patra et al., 2015) NA,3-month Same Italian Sentipolc-16 (Barbieri et al., 2016) 2013-2016 2016 Portug.”