SemEval-2016 Task 6: Detecting Stance in Tweets

By Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiao-Dan Zhu, Colin Cherry. 2016

1,038 citations recorded by the source corpus.

Here for the ﬁrst time we present a shared task on detecting stance from tweets: given a tweet and a target entity (person, organization, etc.), automatic natural language systems must determine whether the tweeter is in favor of the given target, against the given target, or whether neither inference is likely. The target of interest may or may not be referred to in the tweet, and it may or may not be the target of opinion. Two tasks are proposed. Task A is a traditional supervised classiﬁcation task where 70% of the annotated data for a target is used as training and the rest for testing. For Task B, we use as test data all of the instances for a new target (not used in task A) and no training data is provided. Our shared task received submissions from 19 teams for Task A and from 9 teams for Task B. The highest clas-siﬁcation F-score obtained was 67.82 for Task A and 56.28 for Task B. However, systems found it markedly more difﬁcult to infer stance towards the target of interest from tweets that express opinion towards another entity.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, background): “– TweetEval: Emotion Recognition (Mohammad et al., 2018), Emoji Prediction (Barbieri et al., 2018), Irony Detection (Van Hee et al., 2018), Hate Speech Detection (Basile et al., 2019), Offensive Language Identification (Zampieri et al., 2019), Sentiment Analysis (Rosenthal et al., 2017), Stance Dete”
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021, background): “SemEval 2016 Task 6 (Mohammad et al., 2016; Barbieri et al., 2020) This task classifies a tweet’s stance as {“neutral", “against", “favor"}.”
TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification (2020, methodology): “We use the SemEval2016 shared task on Detecting Stance in Tweets (Mohammad et al., 2016).”
FANG: Leveraging Social Context for Fake News Detection Using Graph Representation (2020, background): “Popular stance detection datasets either do not explicitly describe the target text [8], have a limited number of targets [27, 40], or define the source/target texts differently, as in the Fake News Challenge.”
Can Large Language Models Transform Computational Social Science? (2023, methodology): “We evaluate stance detection on the earliest and most established SemEval-2016 Stance Dataset (Mohammad et al., 2016), which contains 1,250 tweets and their associated stance towards five topics: atheism, climate change, the feminist movement, Hillary Clinton, and the legalization of abortion.”
Author’s Sentiment Prediction (2020, background): “In stance detection the models are looking for favorability towards a given target of interest which may not be mentioned or targeted in the text (Mohammad et al., 2016; Augenstein et al., 2016).”
How would Stance Detection Techniques Evolve after the Launch of ChatGPT? (2022, methodology): “Results: To compare the effectiveness of Chat-GPT, we carried out experimental validations in the SemEval-2016 stance dataset(Mohammad et al., 2016) and P-Stance dataset(Li et al., 2021).”
X -stance: A Multilingual Multi-Target Dataset for Stance Detection (2020, methodology): “It seems that the supervised part of X -stance has a similar difﬁculty as the SemEval-2016 (Mohammad et al., 2016a) or MPCHI (Sen et al., 2018) datasets on which B ERT has previously been evaluated.”
ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning (2021, background): “Stance Prediction and Argumentation: Previous stance prediction works have been largely applied to online content, for political, ideological debates, rumor and fake news detection (Mohammad et al., 2016; Derczynski et al., 2017; Hardalov et al., 2021).”
ConStance: Modeling Annotation Contexts to Improve Stance Classification (2017, methodology): “Classiﬁer performance on the GS is measured, following prior work (Mohammad et al., 2016a; Ebrahimi et al., 2016), on the average of the F1 scores on the two classes of interest (“Clinton” and “Trump”).”
SemEval-2017 Task 4: Sentiment Analysis in Twitter (2017, background): “…sentiment analysis of figurative language on Twitter (Ghosh et al., 2015), implicit event polarity (Russo et al., 2015), detecting stance in tweets (Mohammad et al., 2016a), out-ofcontext sentiment intensity of words and phrases (Kiritchenko et al., 2016), and emotion detection (Strapparava and…”
Meta-tuning Language Models to Answer Prompts Better (2021, background): “If you can #dream it, you can #make it so " SemEval 2016 Task 6 (Mohammad et al., 2016; Barbieri et al., 2020) This task classiﬁes a tweet’s stance as {“neutral", “against", “favor"}.”
TimeLMs: Diachronic Language Models from Twitter (2022, background): “…et al., 2018), emotion recognition (Mohammad et al., 2018), hate speech detection (Basile et al., 2019), irony detection (Van Hee et al., 2018), offensive language identiﬁcation (Zampieri et al., 2019), sentiment analysis (Rosenthal et al., 2017) and stance detection (Mohammad et al., 2016).”
Sentiment Analysis in the Era of Large Language Models: A Reality Check (2023, background): “MAST ABSA on Detection Stance in Tweets (Mohammad et al., 2016) for stance detection task; (6) CS19 dataset (Panchenko et al., 2019) for comparative opinion mining task; (7) TweetEval benchmark (Barbieri et al., 2020) for emotion recognition task.”
Cross-topic Argument Mining from Heterogeneous Sources (2018, background): “4 The second corpus, from SemEval-2016 Task 6 (Mohammad et al., 2016), consists of around”