Can AI-Generated Text be Reliably Detected?

By Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Soheil Feizi. 2023

604 citations recorded by the source corpus.

Large Language Models (LLMs) perform impressively well in various applications. However, the potential for misuse of these models in activities such as plagiarism, generating fake news, and spamming has raised concern about their responsible use. Consequently, the reliable detection of AI-generated text has become a critical area of research. AI text detectors have shown to be effective under their specific settings. In this paper, we stress-test the robustness of these AI text detectors in the presence of an attacker. We introduce recursive paraphrasing attack to stress test a wide range of detection schemes, including the ones using the watermarking as well as neural network-based detectors, zero shot classifiers, and retrieval-based detectors. Our experiments conducted on passages, each approximately 300 tokens long, reveal the varying sensitivities of these detectors to our attacks. Our findings indicate that while our recursive paraphrasing method can significantly reduce detection rates, it only slightly degrades text quality in many cases, highlighting potential vulnerabilities in current detection systems in the presence of an attacker. Additionally, we investigate the susceptibility of watermarked LLMs to spoofing attacks aimed at misclassifying human-written text as AI-generated. We demonstrate that an attacker can infer hidden AI text signatures without white-box access to the detection method, potentially leading to reputational risks for LLM developers. Finally, we provide a theoretical framework connecting the AUROC of the best possible detector to the Total Variation distance between human and AI text distributions. This analysis offers insights into the fundamental challenges of reliable detection as language models continue to advance. Our code is publicly available at https://github.com/vinusankars/Reliability-of-AI-text-detectors.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, neutral): “However, technical tools such as detection are less accurate as AI systems become more powerful [204].”
AI deception: A survey of examples, risks, and potential solutions (2023, neutral): “85 Attackers will attempt to bypass these defenses, 86 but companies should be required to stay ahead of these attacks and provide trustworthy techniques for identifying AI outputs.”
GPT detectors are biased against non-native English writers (2023, background): “Our investigation into the effectiveness of simple prompts in bypassing GPT detectors, along with recent studies on paraphrasing attacks (Krishna et al., 2023; Sadasivan et al., 2023), raises significant concerns about the reliability of current detection methods.”
Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text (2024, neutral): “A string of characters s can be parsed into tokens and represented as a list of token indices ⃗x by a tokenizer T .”
DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature (2023, methodology): “However, these mask-ﬁlling models are used off-the-shelf, without any ﬁne-tuning or adaptation to the target domain.”
ChatGPT: The End of Online Exam Integrity? (2022, neutral): “The landscape is further complicated by adversarial training techniques, which allow LLMs to adapt and learn patterns that specifically avoid detection, thus challenging the efficacy of even the most advanced machine learning-based detectors [43].”
DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text (2023, background): “The detection of machine-generated text is commonly formulated as a classification task (Jawahar et al., 2020; Fagni et al., 2021; Bakhtin et al., 2019; Sadasivan et al., 2023; Wang et al., 2023).”
Breaking Distortion-free Watermarks in Large Language Models (2025, unknown): “The first work that comprehensively studied spoofing is Sadasivan et al. [24], which targets Kirchenbauer et al. [12]’s distribution-modifying watermark.”
Understanding Generative AI Content with Embedding Models (2024, background): “However, [23] provides evidence that these popular detection tools are easily fooled with clever prompting techniques.”