Typical Decoding for Natural Language Generation

By Clara Meister, Tiago Pimentel, Gian Wiher, Ryan Cotterell. 2022

52 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, background): “…sampling (Ackley et al., 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al., 2020), minimum Bayes risk decoding (Eikema & Aziz, 2020; Shi et al., 2022), and typical decoding (Meister et al., 2022).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “introduced recently to improve the quality of LMgenerated text (Graves, 2012; Fan et al., 2018; Holtzman et al., 2020; Meister et al., 2022).”
RankGen: Improving Text Generation with Large Ranking Models (2022, methodology): “Decoding algorithms: For each LM considered we decode outputs using greedy decoding, ancestral sampling, nucleus sampling (Holtzman et al., 2020), top-k sampling (Fan et al., 2018), and typical sampling (Meister et al., 2022).”
Language Models Can See: Plugging Visual Controls in Text Generation (2022, methodology): “Following previous studies [83, 49, 72], we ﬁrst evaluate the quality of the generated results from different methods using automatic evaluation metrics, including (1) n -gram repetition (rep-n ); (2) generation diversity (div.); (3) semantic coherence (coh.) between the generated story and the…”
Offline RL for Natural Language Generation with Implicit Language Q Learning (2022, background): “Much prior work on steering language models towards desired behavior has done so without an explicit utility function, instead either carefully curating finetuning datasets [32, 33, 34], or designing subtle decoding heuristics [35, 36, 37, 38, 39, 40, 41, 42].”
Differentially Private Decoding in Large Language Models (2022, background): “Exploring such strategies is still a problem of interest to the community (Meister et al., 2022; Holtzman et al., 2019).”
Contrastive Decoding: Open-ended Text Generation as Optimization (2022, methodology): “Prior work has found that these methods outperform other proposed decoding algorithms (DeLucia et al., 2020; Meister et al., 2022) small medium large XL student small”
Gradient-based Constrained Sampling from Language Models (2022, background): “In future work, we will explore constraints designed to discourage this behavior (Welleck et al., 2020; Meister et al., 2022).”