Emergent: a novel data-set for stance classification

By William Ferreira, Andreas Vlachos. 2016

410 citations recorded by the source corpus.

We present Emergent, a novel data-set derived from a digital journalism project for rumour debunking. The data-set contains 300 rumoured claims and 2,595 associated news articles, collected and labelled by journalists with an estimation of their veracity (true, false or unverified). Each associated article is summarized into a headline and labelled to indicate whether its stance is for, against, or observing the claim, where observing indicates that the article merely repeats the claim. Thus, Emergent provides a real-world data source for a variety of natural language processing tasks in the context of fact-checking. Further to presenting the dataset, we address the task of determining the article headline stance with respect to the claim. For this purpose we use a logistic regression classifier and develop features that examine the headline and its agreement with the claim. The accuracy achieved was 73% which is 26% higher than the one achieved by the Excitement Open Platform (Magnini et al., 2014).

Subfields: Artificial Intelligence, Information Systems, Sociology and Political Science, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Sociology and Political Science

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Reasoning Over Semantic-Level Graph for Fact Checking (2019, background): “POLIFACT is further investigated by following works, including Ferreira and Vlachos (2016) who build Emergent with 300 labeled rumors and about 2.6K news articles, Wang (2017) who builds LIAR with 12.8K annotated short statements and six ﬁne-grained labels, and Rashkin et al. (2017) who collect…”
“Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection (2017, background): “Recently, Ferreira and Vlachos (2016) have released the Emergent dataset, which includes 300 labeled rumors from PolitiFact.”
Explainable Automated Fact-Checking for Public Health Claims (2020, background): “A number of recent works in automated factchecking look at various formulations of factchecking and its analogous tasks (Ferreira and Vlachos, 2016; Hassan et al., 2017; Zlatkova et al., 2019).”
Automated Fact Checking: Task Formulations, Methods and Future Directions (2018, methodology): “The Fake News Challenge (Pomerleau and Rao, 2017) also formulated this part of the fact checking process in the same way, but in contrast to (Ferreira and Vlachos, 2016), entire documents are used as evidence, thus allowing for evidence from multiple sentences to be combined.”
Understanding the Promise and Limits of Automated Fact-Checking (2018, background): “A crucial element in strengthening such approaches, and one which can also be used to assist human factcheckers, is stance detection: determining whether a particular document supports the claim in question (see Ferreira and Vlachos 2016).”
MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims (2019, background): “to fact checking, are stance detection for claims (Ferreira and Vlachos, 2016; Pomerleau and Rao, 2017; Augenstein et al., 2016a; Kochkina et al., 2017; Augenstein et al., 2016b; Zubiaga et al., 2018; Riedel et al., 2017), satire detection (Rubin et al.”
Fighting an Infodemic: COVID-19 Fake News Dataset (2020, methodology): “For rumour debunking, (Ferreira and Vlachos, 2016) created a dataset named Emergent , which is derived from a digital journalism project.”
Evidence-based Verification for Real World Information Needs (2021, background): “Stance Classification vs Evidence Extraction Pipeline Stance classification for fact checking (Ferreira and Vlachos, 2016; Karadzhov et al., 2017) assumes the evidence E is provided as a passage of text P = E.”
Stance Detection on Social Media: State of the Art and Trends (2020, methodology): “Additionally, the Emergent dataset [40] is a Claim-based stance detection dataset for Fact checking.”
TwoWingOS: A Two-Wing Optimization Strategy for Evidential Claim Verification (2018, background): “Ferreira and Vlachos (2016) release the dataset “Emergent” for rumor debunking.”
Zero-Shot Stance Detection: A Dataset and Model Using Generalized Topic Representations (2020, background): “, 2018; Ferreira and Vlachos, 2016). In a similar vein, Habernal et al. (2018) use comments from news articles and manually construct position statements.”
Fact or Fiction: Verifying Scientific Claims (2020, methodology): “Fact checking and rationalized NLP models Fact-checking datasets include PolitiFact (Vlachos and Riedel, 2014), Emergent (Ferreira and Vlachos, 2016), LIAR (Wang, 2017), SemEval 2017 Task 8 RumorEval (Derczynski et al.”
Preserving integrity in online social networks (2020, background): “Ferreira and Vlachos [35] use article headlines as evidence to predict whether an article is supporting, refuting or observing a claim.”
A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking (2019, methodology): “Emergent16 A more comprehensive corpus for automated fact-checking was introduced by Ferreira and Vlachos (2016). The dataset is based on the project Emergent9 which is a journalist initiative for rumor debunking.”