Comparison of Diverse Decoding Methods from Conditional Language Models

By Daphne Ippolito, Reno Kriz, Maria Kustikova, João Sedoc, Chris Callison-Burch. 2019

135 citations recorded by the source corpus.

While conditional language models have greatly improved in their ability to output high quality natural language, many NLP applications benefit from being able to generate a diverse set of candidate sequences. Diverse decoding strategies aim to, within a given-sized candidate list, cover as much of the space of high-quality outputs as possible, leading to improvements for tasks that rerank and combine candidate outputs. Standard decoding methods, such as beam search, optimize for generating high likelihood sequences rather than diverse ones, though recent work has focused on increasing diversity in these methods. In this work, we perform an extensive survey of decoding-time strategies for generating diverse outputs from a conditional language model. In addition, we present a novel method where we over-sample candidates, then use clustering to remove similar sequences, thus achieving high diversity without sacrificing quality.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Towards a Human-like Open-Domain Chatbot (2020, background): “Many works, e.g., Zhao et al. (2017); Xu et al. (2018); Ippolito et al. (2019b), have focused solely on the diversity aspect to counter the commonly observed problem of models generating generic responses (Li et al., 2016a).”
ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, background): “…decoding strategies have advantages in terms of synthesis diversity, deterministic decoding strategies ( e . g ., beam search and its variants) are more suitable for cases where there is less tolerance for synthesis errors and more emphasis on fluency and coherence (Ippolito et al., 2019).”
Trading Off Diversity and Quality in Natural Language Generation (2020, methodology): “The concept of oversampling generations and then ranking them has been popular since the days of statistical machine translation (Shen et al., 2004) but has also been used more recently in other domains (Li et al., 2016; Ippolito et al., 2019b; Kriz et al., 2019).”
Towards Making the Most of ChatGPT for Machine Translation (2023, background): “Basically, decoding with higher temperatures displays greater linguistic variety, while the low one generates grammatically correct and deterministic text (Ippolito et al., 2019).”
Advances and Challenges in Conversational Recommender Systems: A Survey (2021, background): “Thus, other metrics reflecting user satisfaction are more suitable in evaluation, such as measuring fluency [17, 142, 46], consistency [53, 96], readability [95], informativeness [74], diversity [104, 78, 57], and empathy [58, 175].”
NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics (2021, background): “The rise of autoregressive language models like GPT (Radford et al., 2018) has inspired a ﬂurry of work on decoding strategies (Post and Vilar, 2018a; Ippolito et al., 2019; Zheng et al., 2020; Leblond et al., 2021; West et al., 2021).”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, methodology): “In these cases, we adjust the generation randomness by increasing the decoding temperature [33, 34, 35] from 0 (deterministic mode) to 0 .”
COD3S: Diverse Generation with Discrete Semantic Signatures (2020, background): “We refer readers to Ippolito et al. (2019) for an overview of diverse decoding methods.”