SemEval-2019 Task 4: Hyperpartisan News Detection

By Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel M. Vincent, Payam Adineh, David P. A. Corney, Benno Stein, Martin Potthast. 2019

272 citations recorded by the source corpus.

Hyperpartisan news is news that takes an extreme left-wing or right-wing standpoint. If one is able to reliably compute this meta information, news articles may be automatically tagged, this way encouraging or discouraging readers to consume the text. It is an open question how successfully hyperpartisan news detection can be automated, and the goal of this SemEval task was to shed light on the state of the art. We developed new resources for this purpose, including a manually labeled dataset with 1,273 articles, and a second dataset with 754,000 articles, labeled via distant supervision. The interest of the research community in our task exceeded all our expectations: The datasets were downloaded about 1,000 times, 322 teams registered, of which 184 configured a virtual machine on our shared task cloud service TIRA, of which in turn 42 teams submitted a valid run. The best team achieved an accuracy of 0.822 on a balanced sample (yes : no hyperpartisan) drawn from the manually tagged corpus; an ensemble of the submitted systems increased the accuracy by 0.048.

Subfields: Artificial Intelligence, Sociology and Political Science, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Big Bird: Transformers for Longer Sequences (2020, neutral): “Model IMDb [65] Yelp-5 [108] Arxiv [36] Patents [54] Hyperpartisan [48] # Examples 25000 650000 30043 1890093 645 # Classes 2 5 11 663 2 Excess fraction 0.”
Longformer: The Long-Document Transformer (2020, methodology): “, 2011) and Hyperpartisan news detection (Kiesel et al., 2019) datasets.”
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020, background): “For HYPERPARTISAN, LONGFORMER, a modified Transformer language model for long documents (Beltagy et al., 2020).”
Muppet: Massive Multi-task Representations with Pre-Finetuning (2021, methodology): “For each of the individual downstream tasks, we use a ﬁxed hyper-parameter search to optimize over simple hyperparameters such as learning rate, Adam (cid:15) (Kingma and Ba, 2014) and dropout (Sri-vastava et al., 2014).”
ELLE: Efficient Lifelong Pre-training for Emerging Data (2022, methodology): “PER P ARTISAN (Kiesel et al., 2019), H ELPFULLNESS (McAuley et al., 2015), C HEM P ROT (Kringelum et al., 2016) and ACL-ARC (Jurgens et al., 2018) for WB, N S , R EV , B IO and CS, respectively.”
Recurrent Memory Transformer (2022, methodology): “Moreover, a combination of recurrent memory with RoBERTa-base results in state of the art performance for the Hyperpartisan news classification task (Kiesel et al., 2019).”
Discovering Biased News Articles Leveraging Multiple Human Annotations (2020, methodology): “6 Additionally to these humanly labeled articles ( E and NE ), we use the training data given to the participants of the Semeval 2019 task for hyperpartisanship detection (denoted as P ) to compare our performance against it (Kiesel et al., 2019).”
SMYRF: Efficient Attention using Asymmetric Clustering (2020, methodology): “To compare with the Longformer [28], we evaluate SMYRF on the Hyperpartisan News Detection [55] dataset.”
Data Selection for Language Models via Importance Resampling (2023, background): “Following Gururangan et al. [24], we consider 8 downstream datasets across 4 domains: Computer Science papers (ACL-ARC [31], Sci-ERC [51]), Biomedicine (ChemProt [41], RCT [16]) News (AGNews [92], HyperPartisan [34]), and Reviews (Helpfulness [53], IMDB [52]).”
Team Bertha von Suttner at SemEval-2019 Task 4: Hyperpartisan News Detection using ELMo Sentence Representation Convolutional Network (2019, background): “SemEval-2019 Task 4 on “Hyperpartisan News Detection” (Kiesel et al., 2019) is a document-level classiﬁcation task which requires building a precise and reliable algorithm to automatically discriminate hy-perpartisan news from more balanced stories.”
Revisiting Transformer-based Models for Long Document Classification (2022, methodology): “3 To address the generalisation concern, we also use three datasets from other domains: EC-tHR (Chalkidis et al., 2022) sourced from legal cases, Hyperpartisan (Kiesel et al., 2019) and 20 News (Joachims, 1997), both from news articles.”
Black-box Prompt Learning for Pre-trained Language Models (2022, background): “, 2018), RCT (Dernoncourt and Lee, 2017), HYPERPARTISAN (Kiesel et al., 2019) from specific domains including computer science, biomedical science and news following Gururangan et al.”
Assessing Political Prudence of Open-domain Chatbots (2021, methodology): “In this work, we use a BERT-based binary classiﬁer ﬁnetuned on hyper-partisan news classiﬁcation task data (Kiesel et al., 2019), following the method Lee et al. (2019a),which achieved a 76 .”