SemEval-2013 Task 2: Sentiment Analysis in Twitter

By Preslav Nakov, Sara Rosenthal, Zornitsa Kozareva, Veselin Stoyanov, Alan Ritter, Theresa Wilson. 2013

410 citations recorded by the source corpus.

In recent years, sentiment analysis in social media has attracted a lot of research interest and has been used for a number of applications. Unfortunately, research has been hindered by the lack of suitable datasets, complicating the comparison between approaches. To address this issue, we have proposed SemEval-2013 Task 2: Sentiment Analysis in Twitter, which included two subtasks: A, an expression-level subtask, and B, a messagelevel subtask. We used crowdsourcing on Amazon Mechanical Turk to label a large Twitter training dataset along with additional test sets of Twitter and SMS messages for both subtasks. All datasets used in the evaluation are released to the research community. The task attracted significant interest and a total of 149 submissions from 44 teams. The bestperforming team achieved an F1 of 88.9% and 69% for subtasks A and B, respectively.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021, background): “, 2011)), improving it on Twitter data (Senti140 and SemEval 2017 (Go et al., 2009; Nakov et al., 2013)) and contrast sets (Gardner et al.”
Like It or Not (2016, methodology): “This approach has been used for manual annotation of several datasets [Go et al. 2009; Shamma et al. 2009; Nakov et al. 2013].”
Sentiment Embeddings with Applications to Sentiment Analysis (2016, background): “Sentence level sentiment classiﬁcation on tweets [21], [22] and reviews [23] help us understand whether sentiment embeddings are helpful in capturing discriminative features for predict the sentiment of text.”
Sentiment Analysis of Short Informal Texts (2014, methodology): “In this work, we follow the deﬁnition of the task and use the data provided for the SemEval-2013 competition: Sentiment Analysis in Twitter (Wilson et al., 2013).”
Deep Convolutional Neural Networks for Sentiment Analysis of Short Texts (2014, background): “The advent of online social networks has produced a crescent interest on the task of sentiment analysis for short text messages (Go et al., 2009; Barbosa and Feng, 2010; Nakov et al., 2013).”
SemEval-2015 Task 10: Sentiment Analysis in Twitter (2015, background): “One such annotation scheme is MaxDiff (Lou-viere, 1991), which is widely used in market surveys (Almquist and Lee, 2009); it was also used in a previous SemEval task (Jurgens et al., 2012).”
SemEval-2017 Task 5: Fine-Grained Sentiment Analysis on Financial Microblogs and News (2017, background): “The TrendMiner (2011-2014) FP7 European project 29 , presents an innovative and portable open-source real-time method for cross-lingual mining and summarisation of large-scale social media streams, such as weblogs, Twitter, Facebook, etc.”
Predicting Polarities of Tweets by Composing Word Embeddings with Long Short-Term Memory (2015, methodology): “Different from STS dataset deciding the polarity based on emoticons, the benchmark dataset in SemEval 2013 (Nakov et al., 2013) is labelled by human annotators.”
Stance and Sentiment in Tweets (2016, methodology): “12A similar metric was used in the past for sentiment analysis—SemEval 2013 Task 2 [Wilson et al. 2013].”
Gated Neural Networks for Targeted Sentiment Analysis (2016, methodology): “Targeted sentiment analysis is related to fine-grained sentiment analysis (Wiebe, Wilson, and Cardie 2005; Jin, Ho, and Srihari 2009; Li et al. 2010; Yang and Cardie 2013; Nakov et al. 2013), which extracts opinion expressions, holders and targets jointly from given sentences.”
Fine-tune BERT with Sparse Self-Attention Mechanism (2019, methodology): “We adopt the dataset version provided by GLUE (Wang et al., 2018); 3) SemEval: The Se-mEval 2013 Twitter dataset (Nakov et al., 2013) contains tweets with three classes: positive, negative, and neutral; 4) Sentube-A, Sentube-T: The SenTube datasets (Uryupina et al., 2014) are texts obtained from…”
Improving Hate Speech Detection with Deep Learning Ensembles (2018, methodology): “We evaluate our method on two Twitter classiﬁcation datasets, abusive speech (Waseem and Hovy, 2016) and SemEval 2013 sentiment analysis (Nakov et al., 2013) dataset (Table 1).”
Ultradense Word Embeddings by Orthogonal Transformation (2016, methodology): “We follow the ofﬁcial guidelines and use the Sem-Eval2013 training and development sets as training set, the SemEval2013 test set as development set and the SemEval2015 test set to report ﬁnal scores (Nakov et al., 2013; Rosenthal et al., 2015).”
Twitter Sentiment Analysis (2015, methodology): “B (Nakov et al., 2013) (see Figure 2b).”
NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets (2013, methodology): “We developed these classifiers to participate in an international competition organized by the Conference on Semantic Evaluation Exercises (SemEval-2013) (Wilson et al., 2013).”
SemEval-2016 Task 5: Aspect Based Sentiment Analysis (2016, background): “…and conferences focus-19 ing on somewhat different research problems (Seki et al., 2007; Seki et al., 2008; Seki et al., 2010; Mitchell, 2013; Nakov et al., 2013; Rosenthal et al., 2014; Pontiki et al., 2014; Rosenthal et al., 2015; Ghosh et al., 2015; Pontiki et al., 2015; Moham-mad et al.,…”