A Contrastive Framework for Neural Text Generation

By Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, Nigel Collier. Neural Information Processing Systems, 2022

334 citations recorded by the source corpus.

Text generation is of great importance to many natural language processing applications. However, maximization-based decoding methods (e.g. beam search) of neural language models often lead to degenerate solutions -- the generated text is unnatural and contains undesirable repetitions. Existing approaches introduce stochasticity via sampling or modify training objectives to decrease probabilities of certain tokens (e.g., unlikelihood training). However, they often lead to solutions that lack coherence. In this work, we show that an underlying reason for model degeneration is the anisotropic distribution of token representations. We present a contrastive solution: (i) SimCTG, a contrastive training objective to calibrate the model's representation space, and (ii) a decoding method -- contrastive search -- to encourage diversity while maintaining coherence in the generated text. Extensive experiments and analyses on three benchmarks from two languages demonstrate that our proposed approach significantly outperforms current state-of-the-art text generation methods as evaluated by both human and automatic metrics.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/c1ffa773f29e567e13b00495767f8c42e2bb44b2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Quark: Controllable Text Generation with Reinforced Unlearning (2022, methodology): “Baselines results are adopted from [72].”
RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment (2023, methodology): “As an extension, we may consider more advanced search methods, including the beam search (Reddy, 1977), top-k sampling (Fan et al., 2018), top-p sampling (Holtzman et al., 2019), contrastive search (Su et al., 2022).”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “Contrastive search [226] further induces a degeneration term to penalize the generation of the word that has similar meaning with previous context.”
Contrastive Decoding: Open-ended Text Generation as Optimization (2022, methodology): “org greedy (max prob) decoding that uses log pEXP as the objective, and contrastive search (CS) (Su et al., 2022; Su and Collier, 2022).”
RankGen: Improving Text Generation with Large Ranking Models (2022, methodology): “…al., 2018; Qi et al., 2020), rely on retrieval (Khandelwal et al., 2020), use bidirectional LMs (Serdyuk et al., 2018), employ contrastive learning (Su et al., 2022; An et al., 2022), use BERT for sentence-level language modeling (Ip-polito et al., 2020), and designing sequence-level losses…”
Language Models Can See: Plugging Visual Controls in Text Generation (2022, methodology): “We use a publicly available GPT-based language model [72] which is finetuned on the ROCStories benchmark.”
Data Feedback Loops: Model-driven Amplification of Dataset Biases (2022, methodology): “We measure two quality metrics and one generalization metric: 1) coherence score (Su et al., 2022), defined as the average similarity between prompts and corresponding model completions featurized by a sentence embedding model; 2) mauve score (Pillutla et al.”