Automatic Detection of Generated Text is Easiest when Humans are Fooled

By Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, Douglas Eck. 2019

524 citations recorded by the source corpus.

Recent advancements in neural language modelling make it possible to rapidly generate vast amounts of human-sounding text. The capabilities of humans and automatic discriminators to detect machine-generated text have been a large source of research interest, but humans and machines rely on different cues to make their decisions. Here, we perform careful benchmarking and analysis of three popular sampling-based decoding strategies—top-_k_, nucleus sampling, and untruncated random sampling—and show that improvements in decoding methods have primarily optimized for fooling humans. This comes at the expense of introducing statistical abnormalities that make detection easy for automatic systems. We also show that though both human and automatic detector performance improve with longer excerpt length, even multi-sentence excerpts can fool expert human raters over 30% of the time. Our findings reveal the importance of using both human and automatic detectors to assess the humanness of text generation systems.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “, 2021) or trained, in-person evaluators (Ippolito et al., 2020) were needed to distinguish between human-authored text and text generated by today’s generation mod-”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “with human-like fluency, coherence, and realism (Zellers et al., 2020; Ippolito et al., 2020) has caused a paradigm shift in our society.”
DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text (2023, background): “…source LLM to extract statistics such as the average per-token log probability or the average rank of each token in the ranked list of possible choices to make a prediction by comparing it to a threshold (Solaiman et al., 2019; Ippolito et al., 2019; Gehrmann et al., 2019; Mitchell et al., 2023).”
Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature (2023, background): “…typically trained using bag-of-words (Solaiman et al., 2019; Fagni et al., 2021) or neural representations (Bakhtin et al., 2019; Solaiman et al., 2019; Uchendu et al., 2020; Ippolito et al., 2020; Fagni et al., 2021; Yan et al., 2023; Pu et al., 2023; Mitrovi ´ c et al., 2023; Li et al., 2023).”
Residual Energy-Based Models for Text (2021, background): “However, discriminating real vs. machine-generated text is an important task on its own, and has recently gained a lot of attention (Gehrmann et al., 2019; Radford et al., 2019; Zellers et al., 2019a; Ippolito et al., 2020).”
Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense (2023, methodology): “Other related studies examine classiﬁcation performance across domains [Bakhtin et al., 2019] and decoding strategies [Ippolito et al., 2020].”
Automatic Detection of Machine Generated Text: A Critical Survey (2020, background): “Differences in human and machine detector : Ippolito et al., (2020) study the differences in the ability of humans and automated detectors to identify text generated by TGM.”
MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers (2021, methodology): “Given the known difficulties with human evaluation of longer texts [28], we use a maximum completion length of 256 tokens.”
Can AI-Generated Text be Reliably Detected? (2023, methodology): “Non-watermarking detectors such as trained classiﬁers [3], retrieval-based detectors [4], and zero-shot classiﬁers [2, 24, 23, 22] use the presence of LLM-speciﬁc signatures in AI-generated texts for their detection.”
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation (2021, background): “Previous studies have asked workers to choose from (Mori et al., 2019) or distinguish between human-written and machine-generated texts (Garbacea et al., 2019; Ippolito et al., 2020; Clark et al., 2021).”
DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature (2023, methodology): “The most common method for zero-shot machine-generated text detection is evaluating the average per-token log probability of the generated text and thresholding (Solaiman et al., 2019; Gehrmann et al., 2019; Ippolito et al., 2020).”
Do Language Models Plagiarize? (2022, methodology): “Particularly, top-p sampling is reported to be the most effective decoding method in generating high-quality texts [23].”
Human heuristics for AI-generated language are flawed (2022, background): “Recent work by Ippolito et al. (35) suggests that language model–decoding methods have been optimized for fooling humans at the cost of introducing statistical anomalies easily detected by machines.”
Testing of detection tools for AI-generated text (2023, background): “Firstly, several researchers has studied human abilities to detect machine-generated texts (e.g. Guo et al., 2023; Ippolito et al., 2020; Ma et al., 2023).”