Defending Against Neural Fake News

By Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, Yejin Choi. 2019

1,265 citations recorded by the source corpus.

Recent progress in natural language generation has raised dual-use concerns. While applications like summarization and translation are positive, the underlying technology also might enable adversaries to generate neural fake news: targeted propaganda that closely mimics the style of real news. Modern computer security relies on careful threat modeling: identifying potential threats and vulnerabilities from an adversary's point of view, and exploring potential mitigations to these threats. Likewise, developing robust defenses against neural fake news requires us first to carefully investigate and characterize the risks of these models. We thus present a model for controllable text generation called Grover. Given a headline like `Link Found Between Vaccines and Autism,' Grover can generate the rest of the article; humans find these generations to be more trustworthy than human-written disinformation. Developing robust verification techniques against generators like Grover is critical. We find that best current discriminators can classify neural fake news from real, human-written, news with 73% accuracy, assuming access to a moderate level of training data. Counterintuitively, the best defense against Grover turns out to be Grover itself, with 92% accuracy, demonstrating the importance of public release of strong generators. We investigate these results further, showing that exposure bias -- and sampling strategies that alleviate its effects -- both leave artifacts that similar discriminators can pick up on. We conclude by discussing ethical issues regarding the technology, and plan to release Grover publicly, helping pave the way for better detection of neural fake news.

Subfields: Artificial Intelligence, Sociology and Political Science, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “, 2013], as a pre-training dataset [Baevski et al., 2019; Zellers et al., 2019; Liu et al., 2019c], and even simply as a giant text corpus for testing optimizers [Anil et al.”
RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, background): “Several efforts have trained on datasets larger and more diverse than the original BERT (Radford et al., 2019; Yang et al., 2019; Zellers et al., 2019).”
Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “We observe that only HellaSwag and MMLU-Humanities appear to have been boosted due to contamination in the training data, with the 70B model appearing to have gained a greater benefit than the 7B model, as one might expect.”
Experience Grounds Language (2020, methodology): “This disconnect is driven home by studies of bias (Gururangan et al., 2018; Glockner et al., 2018) and techniques like Adversarial Filtering (Zellers et al., 2018, 2019b), which elucidate the biases models exploit in lieu of understanding.”
Reducing Sentiment Bias in Language Models via Counterfactual Evaluation (2019, methodology): “…have been applied not only for representation extraction to support downstream tasks (Peters et al., 2018; Devlin et al., 2019), but are also used for many natural language generation applications (Radford et al., 2019; Solaiman et al., 2019; Zellers et al., 2019; Zhang ♦Denotes equal contribution.”
Automatic Detection of Generated Text is Easiest when Humans are Fooled (2019, methodology): “G ROVER was designed to not only generate convincing news excerpts but to also identify them using a ﬁne-tuned version of the generative model itself (Zellers et al., 2019).”
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “We create an aggregate dataset consisting of Wikipedia (Devlin et al., 2018), CC-Stories (Trinh & Le, 2018), RealNews (Zellers et al., 2019), and OpenWeb-text (Radford et al., 2019).”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “Because these latter language models are trained on far more diverse data than is used in the supervised setting, they demonstrate impressive text generation capabilities (Radford et al., 2019; Zellers et al., 2019).”
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, methodology): “We also included the CC-Stories and RealNews datasets used to train Megatron [63].”
Ethical and social risks of harm from Language Models (2021, background): “Fake news generated by simpler language models were also hard to detect and found to pass as human (Zellers et al., 2020).”
TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “Truthfulness is relevant to many applications including generating news stories (Kreps et al., 2020; Zellers et al., 2019), summarization (Gabriel et al., 2021; Maynez et al., 2020; Stiennon et al., 2020; Wang et al., 2020), conversational dialog (Shuster et al., 2021; Roller et al., 2021), and…”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, background): “Given the potential for misuse and harm, we also echo calls for improving policy around public release of large language models (Zellers et al., 2019; McGuffie and Newhouse, 2020).”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “Gehrmann et al. (2019b) showed that these differences exist even for large pretrained models, a fact that was used by Zellers et al. (2019) to train a classifier that detects generated text.”
Deduplicating Training Data Makes Language Models Better (2021, methodology): “38% of the tokens in the ofﬁcial release of 25k GROVER-Mega outputs 3 are part of verbatim matches in RealNews of at least length 50 .”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, background): “Quantifying the number of documents that are machine-generated is an active area of research (Zellers et al., 2019); our findings motivate further work.”
InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “We include some metadata on the code files and StackOverflow questions/answers directly in our training data to allow attribute-conditioned generation [40, 75] and attribute prediction.”
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “…behaviour (Sheng et al., 2019; Liang et al., 2021; Wallace et al., 2019), generate non-factual information (Maynez et al., 2020; Longpre et al., 2021; Devaraj et al., 2022), and other challenges in deployment and evaluation (Zellers et al., 2019; McGuffie & Newhouse, 2020; Talat et al., 2022).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “mans are easily fooled by machine-generated fake news (Zellers et al., 2020; Ippolito et al., 2020).”