The Limitations of Stylometry for Detecting Machine-Generated Fake News

By Tal Schuster, Roei Schuster, Darsh J. Shah, Regina Barzilay. 2019

148 citations recorded by the source corpus.

Recent developments in neural language models (LMs) have raised concerns about their potential misuse for automatically spreading misinformation. In light of these concerns, several studies have proposed to detect machine-generated fake news by capturing their stylistic differences from human-written text. These approaches, broadly termed stylometry, have found success in source attribution and misinformation detection in human-written texts. However, in this work, we show that stylometry is limited against machine-generated misinformation. Whereas humans speak differently when trying to deceive, LMs generate stylistically consistent text, regardless of underlying motive. Thus, though stylometry can successfully prevent impersonation by identifying text provenance, it fails to distinguish legitimate LM applications from those that introduce false information. We create two benchmarks demonstrating the stylistic similarity between malicious and legitimate uses of LMs, utilized in auto-completion and editing-assistance settings.1 Our findings highlight the need for non-stylometry approaches in detecting machine-generated misinformation, and open up the discussion on the desired evaluation benchmarks.

Subfields: Artificial Intelligence, Sociology and Political Science, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Automatic Detection of Generated Text is Easiest when Humans are Fooled (2019, background): “Schuster et al. (2019) argue that the language distributional features implicitly or explicitly employed by these detectors are insufficient; instead, one should look to explicit fact-verification models.”
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence (2021, background): “…texts that match given facts is a known challenge (Fan et al., 2020; Kryscinski et al., 2020; Lewis et al., 2020b; Parikh et al., 2020; Shah et al., 2020; Tian et al., 2020) as language models tend to degenerate and hallucinate (Holtzman et al., 2020; Schuster et al., 2020; Zhou et al., 2020).”
FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information (2021, background): “Without evidence, the ability to detect machine-generated misinformation is inherently limited [Schuster et al., 2020].”
Human and Automatic Detection of Generated Text (2019, background): “Finally, Schuster et al. (2019) argue that language distributional features implicitly or explicitly used by these detectors are insuf-ﬁcient; instead, one should look to explicit fact-veriﬁcation models.”
Exploring AI Ethics of ChatGPT: A Diagnostic Analysis (2023, background): “While studies have concluded that LLMs can be used for social good in real-world applications [12], the vulnerabilities described above can be exploited unethically for unfair discrimination, automated misinformation, and illegitimate censorship [13].”
Evidence-based Verification for Real World Information Needs (2021, background): “Without evidence, the task signature reduces to sentence classiﬁcation where linguistic features alone provide an indication to a claim’s ve-racity, which has limitations (Schuster et al., 2020).”
Automatic Detection of Machine Generated Text: A Critical Survey (2020, background): “Schuster et al., (2020) show that existing detectors perform poorly in these two scenarios as they rely too much on distributional features, which cannot help in distinguishing texts from similar sources.”
Mobile Game Evaluation Method Based on Data Mining of Affective Time Series (2025, unknown): “The idea of combining quantitative (direct measurements, e.g., physiological) and qualitative data (interviews, observations, and questionnaires) in the analysis of players’ emotions is not new.”