SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours

By Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, Arkaitz Zubiaga. International Workshop on Semantic Evaluation, 2017

377 citations recorded by the source corpus.

Media is full of false claims. Even Oxford Dictionaries named “post-truth” as the word of 2016. This makes it more important than ever to build systems that can identify the veracity of a story, and the nature of the discourse around it. RumourEval is a SemEval shared task that aims to identify and handle rumours and reactions to them, in text. We present an annotation scheme, a large dataset covering multiple topics – each having their own families of claims and replies – and use these to pose two concrete challenges as well as the results achieved by participants on these challenges.

Subfields: Artificial Intelligence, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/ca61236a678d6cc280c12ee6bb02175f5221c831

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

Citations in the corpus

DTCA: Decision Tree-based Co-Attention Networks for Explainable Claim Verification (2020, methodology): “We observe that: • SVM integrating semantics from claim content and comments outperforms traditional neural networks only capturing semantics from claim content, like CNN and TE, with at least 4.75% and 6.96% boost in accuracy on the two datasets respectively, which indicates that semantics of comme”
Fact or Fiction: Verifying Scientific Claims (2020, methodology): “Fact-checking datasets include PolitiFact (Vla-chos and Riedel, 2014), Emergent (Ferreira and Vlachos, 2016), LIAR (Wang, 2017), SemEval 2017 Task 8 RumorEval (Derczynski et al., 2017), Snopes (Popat et al., 2017), CLEF-2018 Check-That!”
ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning (2021, background): “Stance Prediction and Argumentation: Previous stance prediction works have been largely applied to online content, for political, ideological debates, rumor and fake news detection (Moham-mad et al., 2016; Derczynski et al., 2017; Hardalov et al., 2021).”
A survey on fake news and rumour detection techniques (2019, methodology): “79 accuracy on the RumorEval dataset [29].”
FANG: Leveraging Social Context for Fake News Detection Using Graph Representation (2020, background): “Popular stance detection datasets either do not explicitly describe the target text [8], have a limited number of targets [27, 40], or define the source/target texts differently, as in the Fake News Challenge.”
Stance Detection on Social Media: State of the Art and Trends (2020, methodology): “In addition, a new wave of stance detection applications has been triggered to handle issues that have infected social media lately, such as fake news and rumors [2, 30, 49], where the stance toward claims in a piece of news is utilized as a key feature for the detection of the credibility of that n”
DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learning (2018, positive): “As the fourth dataset, we consider the benchmark dataset released by SemEval-2017 for the task of determining credibility and stance of social media content (Twitter) (Derczynski et al., 2017).”
Evidence-based Verification for Real World Information Needs (2021, background): “In contrast, MultiFC (Augen-stein et al., 2019) (Derczynski et al., 2017) 330 yes no Twitter (Baly et al., 2018) 442 yes no News + fact checking websites (Thorne and Vlachos, 2018) 185 , 445 no yes Wikipedia (constructed claims) (Chen et al., 2019) 1 , 000 yes no Debate websites (Augenstein et al.,…”
A Stylometric Inquiry into Hyperpartisan and Fake News (2017, background): “For example, Etzioni et al. (2008) propose to identify inconsistencies by matching claims extracted from the web with those of a document in question. Similarly, Magdy and Wanas (2010) measure the frequency of documents that support a claim. Both approaches face the challenges of web data credibilit”
A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking (2019, background): “The corpora introduced by Vla-chos and Riedel (2014); Ferreira and Vlachos (2016); Derczynski et al. (2017) are valuable for the analysis of the fact-checking problem and pro-a vide annotations for stance detection.”
Automated Fact Checking: Task Formulations, Methods and Future Directions (2018, background): “Rumor veracity prediction is the assessment of the macro level behaviors of users’ interactions with and distribution of content to predict whether the claims in the content are true or false (Derczynski and Bontcheva, 2015; Derczynski et al., 2017).”
SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles (2020, methodology): “…and 2019 tasks on Fact Extraction and VERiﬁ-cation (Thorne et al., 2018), the SemEval 2017 and 2019 tasks on determining the veracity of rumors (Derczynski et al., 2017; Gorrell et al., 2019) and the SemEval 2019 task on Fact-Checking in Community Question Answering Forums (Mihaylova et al.,…”
Explainable Automated Fact-Checking: A Survey (2020, background): “, 2020); credibility assessment (Mitra and Gilbert, 2015; Derczynski et al., 2017); and an end-to-end pipeline which encompasses both the document-level evidence retrieval and sentence-level evidence selection sub-tasks of claim verification (Thorne et al.”
Predicting Factuality of Reporting and Bias of News Media Sources (2018, methodology): “…as a task in its own right, where models have been developed based on data from the Fake News Challenge (Riedel et al., 2017; Thorne et al., 2017; Mohtarami et al., 2018; Hanselowski et al., 2018), or from SemEval-2017 Task 8 (Derczynski et al., 2017; Dungs et al., 2018; Zubiaga et al., 2018).”