Neural Text Generation with Unlikelihood Training

By Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, Jason Aaron Edward Weston. 2019

727 citations recorded by the source corpus.

Neural text generation is a key tool in natural language applications, but it is well known there are major problems at its core. In particular, standard likelihood training and decoding leads to dull and repetitive outputs. While some post-hoc fixes have been proposed, in particular top-$k$ and nucleus sampling, they do not address the fact that the token-level probabilities predicted by the model are poor. In this paper we show that the likelihood objective itself is at fault, resulting in a model that assigns too much probability to sequences containing repeats and frequent words, unlike those from the human training distribution. We propose a new objective, unlikelihood training, which forces unlikely generations to be assigned lower probability by the model. We show that both token and sequence level unlikelihood training give less repetitive, less dull text while maintaining perplexity, giving superior generations using standard greedy or beam search. According to human evaluations, our approach with standard beam search also outperforms the currently popular decoding methods of nucleus sampling or beam blocking, thus providing a strong alternative to existing techniques.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “Another pseudo-supervised method is Unlikelihood , which simply optimizes the policy to maximize the probability assigned to y w and minimize the probability assigned to y l ; we use an optional coefﬁcient ↵ 2 [0 , 1] on the ‘unlikelihood’ term. from We also consider PPO [37] using a reward…”
The Curious Case of Neural Text Degeneration (2019, background): “Finally, Welleck et al. (2020) begin to address the problem of neural text degeneration through an “unlikelihood loss”, which decreases training loss on repeated tokens and thus implicitly reduces gradients on frequent tokens as well.”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “The problem of repetition can instead be approached by altering the training objectives, as with coverage mechanisms (See et al., 2017) and context-based losses (Welleck et al., 2019).”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, background): “However, fewer studies have investigated toxicity in autoregressive language models, whose generations also suffer from incoherence, blandness, and repetitiveness (Holtzman et al., 2020; Welleck et al., 2019).”
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, critical): “While some previous work suggested that maximum likelihood training results in repetitive text in model outputs (Welleck et al., 2019) we found this to be rare in our outputs and did not require additional countermeasures to mitigate dis-fluencies.”
OPT: Open Pre-trained Transformer Language Models (2022, methodology): “Future work may wish to incorporate more modern strategies for reducing repetition and improving diversity, such as unlikelihood training (Welleck et al., 2020) or best-ﬁrst decoding (Meis-ter et al., 2020).”
Pretraining Language Models with Human Preferences (2023, methodology): “Unlikelihood Unlikelihood training (Welleck et al., 2020) follows MLE in maximizing the likelihoods of segments exceeding a certain reward threshold t.”
Augmented Language Models: a Survey (2023, background): “LLMs often provide non-factual but seemingly plausible predictions, often referred to as hallucinations (Welleck et al., 2020).”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, background): “Since penalizing only the upcoming token limits prefix strings to only those found in the training distribution, it can cause a distribution mismatch between training sequences and generated sequences (Welleck et al., 2019a).”
Recipes for Safety in Open-domain Chatbots (2020, background): “models susceptible when confronting such language because they will have never seen it before: our models frequently copy the input (Welleck et al., 2020), so they might for example copy the offensive language in the input.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “Altering the behavior of the generation method can dramatically influence human evaluations, while maintaining identical or near-identical perplexity (See et al., 2019; Welleck et al., 2020, 2019; Adiwardana et al., 2020; Roller et al., 2020).”
ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, background): “Training neural networks under MLE does not succeed in modeling sequence probabilities [49], since, at inference, the model is conditioned on sequences that may have never been observed at training time.”
How Decoding Strategies Affect the Verifiability of Generated Text (2019, background): “Given that recent studies (Fan et al., 2018; Holtzman et al., 2019; Welleck et al., 2019) point to repetitiveness as one of the main problems affecting the generation of state-of-the-art models, we mainly consider this dimension in our analysis.”
Red Teaming Language Models with Language Models (2022, background): “Unlikelihood training is effective at mitigating the frequency of repetition in LMgenerated text (Welleck et al., 2020), contradictions in dialogue (Li et al.”
On Faithfulness and Factuality in Abstractive Summarization (2020, background): “…been a growing interest in understanding how maximum likelihood training and approximate beam-search decoding in these models lead to less human-like text in open-ended text generation such as language modeling and story generation (Holtzman et al., 2020; Welleck et al., 2020; See et al., 2019).”
Survey of Hallucination in Natural Language Generation (2022, background): “Some early works [71, 201] focus on the potential pitfalls of utilizing the standard likelihood maximization-based objective in training and decoding of NLG models.”
Recipes for Building an Open-Domain Chatbot (2020, background): “The unlikelihood loss (Welleck et al., 2020; Li et al., 2019a) has been shown to help fix mismatches between human and model distributions across various axes, including decreasing repetitions and mitigating the issue of overrepresented vocabulary tokens.”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “A common issue in text generation is degenerate repetition [21, 23, 51, 71] where generated texts get stuck in repetition loops.”
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2021, background): “Recent work ﬁnds that machine generations can be repetitive and lack diversity (Welleck et al., 2020; Holtzman et al., 2020); one way generated knowledge may differ from human-authored is less creative word choice, diversity, or more repetition.”