How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN

By R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, Asli Celikyilmaz. 2021

180 citations recorded by the source corpus.

Current language models can generate high-quality text. Are they simply copying text they have seen before, or have they learned generalizable linguistic abstractions? To tease apart these possibilities, we introduce RAVEN, a suite of analyses for assessing the novelty of generated text, focusing on sequential structure (n-grams) and syntactic structure. We apply these analyses to four neural language models trained on English (an LSTM, a Transformer, Transformer-XL, and GPT-2). For local structure—e.g., individual dependencies—text generated with a standard sampling scheme is substantially less novel than our baseline of human-generated text from each model’s test set. For larger-scale structure—e.g., overall sentence structure—model-generated text is as novel or even more novel than the human-generated baseline, but models still sometimes copy substantially, in some cases duplicating passages over 1,000 words long from the training set. We also perform extensive manual analysis, finding evidence that GPT-2 uses both compositional and analogical generalization mechanisms and showing that GPT-2’s novel text is usually well-formed morphologically and syntactically but has reasonably frequent semantic issues (e.g., being self-contradictory).

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Quantifying Memorization Across Neural Language Models (2022, background): “McCoy et al. (2021) found that random sampling resulted in generated text with a greater number of novel n -grams.”
Deduplicating Training Data Mitigates Privacy Risks in Language Models (2022, background): “Past and concurrent work uses similar definitions and experimental setups (McCoy et al., 2021; Lee et al., 2021; Carlini et al., 2022).”
Tracing Knowledge in Language Models Back to the Training Data (2022, background): “Carlini et al. (2021) and McCoy et al. (2021) have studied when language models fully replicate passages from their training data.”
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022, background): “There is also a growing body of work analyzing fundamental properties of memorization in language models [16, 47, 60, 95].”
Measuring Causal Effects of Data Statistics on Language Model's 'Factual' Predictions (2022, background): “Similarly, other work has constructed datasets to evaluate the extent of memorization by language models (McCoy et al., 2021; Emami et al., 2020).”
The rise and potential of large language model based agents: a survey (2025, unknown): “The text quality they generate excels in various aspects such as ﬂuency, relevance, diversity, controllability [112, 225–227].”
Do Language Models Plagiarize? (2022, methodology): “Especially, large-scale LMs have demonstrated unprecedented performance on NLG such that LM-generated texts routinely show more novel and interesting stories than human writings do [35], and the distinction between machine-authored and human-written texts has become non-trivial [52, 53].”
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models (2022, background): “Performing only intrinsic evaluation on the training corpora is not sufficient because the intrinsic evaluation itself only tests the capability of the LMs for memorization (McCoy et al., 2021).”
Snoopy: An Online Interface for Exploring the Effect of Pretraining Term Frequencies on Few-Shot LM Performance (2022, background): “works have illustrated language model capabilities to memorize parts of the pretraining data (Carlini et al., 2021; McCoy et al., 2021).”