Real or Fake? Learning to Discriminate Machine from Human Generated Text

By Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc'Aurelio Ranzato, Arthur Szlam. 2019

189 citations recorded by the source corpus.

Energy-based models (EBMs), a.k.a. un-normalized models, have had recent successes in continuous spaces. However, they have not been successfully applied to model text sequences. While decreasing the energy at training samples is straightforward, mining (negative) samples where the energy should be increased is difficult. In part, this is because standard gradient-based methods are not readily applicable when the input is high-dimensional and discrete. Here, we side-step this issue by generating negatives using pre-trained auto-regressive language models. The EBM then works in the residual of the language model; and is trained to discriminate real text from text generated by the auto-regressive models. We investigate the generalization ability of residual EBMs, a pre-requisite for using them in other applications. We extensively analyze generalization for the task of classifying whether an input is machine or human generated, a natural task given the training loss and how we mine negatives. Overall, we observe that EBMs can generalize remarkably well to changes in the architecture of the generators producing negatives. However, EBMs exhibit more sensitivity to the training set used by such generators.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Automatic Detection of Generated Text is Easiest when Humans are Fooled (2019, background): “Bakhtin et al. (2019) frame human-text detection as a ranking task and evaluate their models’ cross-domain and cross-model generalization, finding significant loss in quality when training on one domain and evaluating on another.”
Release Strategies and the Social Impacts of Language Models (2019, background): “Bakhtin and Gross et al. [3] Bakhtin and Gross et al. at Facebook AI Research study detection systems across all three classes.”
Residual Energy-Based Models for Text Generation (2020, methodology): “The book corpus is more challenging because the range of style and topics is more diverse than CC-News.”
The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents (2019, neutral): “He et al. (2019) recently obtained strong results on the DailyDialog and Cornell Movie tasks in terms of perplexity by pre-training on 10% of CC-NEWS (Bakhtin et al., 2019), thus using 100 million sentences (2.7 billion words) and then ﬁne-tuning a transformer based model with a multi-task strategy.”
All-in-One Image-Grounded Conversational Agents (2019, background): “…et al., 2017] have given way to multi-sentence, multilayer, self-attentive representations through Transformers, with pre-training on large corpora such as Wikipedia and Toronto books [Vaswani et al., 2017a; Radford et al., 2018; Devlin et al., 2019; Bakhtin et al., 2019; Radford et al., 2019].”
Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense (2023, methodology): “Other related studies examine classification performance across domains (Bakhtin et al., 2019) and decoding strategies (Ippolito et al.”
Can AI-Generated Text be Reliably Detected? (2023, background): “Several detection works study this problem as a binary classiﬁcation problem [3, 14, 2, 15, 16].”
Mix-review: Alleviate Forgetting in the Pretrain-Finetune Framework for Neural Language Generation Models (2019, unknown): “For pre-training, we use the large-scale CCNEWS data (Bakhtin et al., 2019) which is a deduplicated subset of the English portion of the CommonCrawl news data-set1.”
Analyzing the Forgetting Problem in Pretrain-Finetuning of Open-domain Dialogue Response Models (2021, methodology): “For pretraining, we use the large-scale CCNEWS data (Bakhtin et al., 2019) which is a de-duplicated subset of the English portion of the CommonCrawl news dataset1.”
Human and Automatic Detection of Generated Text (2019, background): “Bakhtin et al. (2019) frame human-text detection as a ranking task and evaluate their models’ cross-domain and cross-model generalization, ﬁnding signiﬁcant loss in quality when training on one domain and evaluating on another.”
Limits of Detecting Text Generated by Large-Scale Language Models (2020, background): “Several recent studies have used supervised learning to develop classifiers for this task [9], [10], [15]–[17] and interpreted their properties.”
DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text (2023, background): “However, models trained explicitly to detect machine-generated texts may overfit to their training distribution of the domains (Bakhtin et al., 2019; Uchendu et al., 2020).”
The Limitations of Stylometry for Detecting Machine-Generated Fake News (2019, background): “Bakhtin et al. (2019) learn a dedicated provenance neural classifier.”
RankGen: Improving Text Generation with Large Ranking Models (2022, background): “…2006) for generative modeling (Grover et al., 2019; Parshakova et al., 2019), speech recognition (Wang and Ou, 2018), open-ended text generation (Bakhtin et al., 2019; Deng et al., 2020), machine translation (Shen et al., 2004; Lee et al., 2021; Bhattacharyya et al., 2021), constrained…”
DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature (2023, background): “However, Bakhtin et al. (2019); Uchendu et al. (2020) note that models trained explicitly to detect machine-generated text tend to overﬁt to their training distribution of data or source models.”
Automatic Detection of Machine Generated Text: A Critical Survey (2020, background): “Existing detectors exhibit poor cross-domain accuracy, that is, they are not generalizable to different publication formats (Wikipedia, books, news sources) (Bakhtin et al., 2019).”
Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature (2023, background): “…often exhibit overfitting tendencies, adapting too closely to the specific distribution of text domains and source models during training (Bakhtin et al., 2019; Uchendu et al., 2020), which consequently leads to limited generalization capabilities when exposed to out-of-distribution data…”