Learning to Write with Cooperative Discriminators

By Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, Yejin Choi. 2018

256 citations recorded by the source corpus.

Despite their local fluency, long-form text generated from RNNs is often generic, repetitive, and even self-contradictory. We propose a unified learning framework that collectively addresses all the above issues by composing a committee of discriminators that can guide a base RNN generator towards more globally coherent generations. More concretely, discriminators each specialize in a different principle of communication, such as Grice’s maxims, and are collectively combined with the base RNN generator through a composite decoding objective. Human evaluation demonstrates that text generated by our model is preferred over that of baselines by a large margin, significantly enhancing the overall coherence, style, and information of the generations.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “…representations (Conneau et al., 2017); moreover, models trained on these datasets were used as components for performing better video captioning (Pasunuru and Bansal, 2017), summarization (Pasunuru and Bansal, 2018), and generation (Holtzman et al., 2018), conﬁrming the importance of NLI research.”
The Curious Case of Neural Text Degeneration (2019, background): “Holtzman et al., 2018; Radford et al., 2019), instead of aiming to decode text that maximizes likelihood.”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, background): “Since models are biased towards copying the input they are conditioned on (Holtzman et al., 2018), our results suggest a possible mechanism for toxicity (or correlates of toxicity) propagation through generation.”
Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “, 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al.”
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022, methodology): “This is similar in spirit to previous work on using cooperative discriminators on uncontrolled LLMs (Holtzman et al., 2018; Krause et al., 2020; Yang and Klein, 2021; Liu et al., 2021a), yet in this work our LLM is controlled in an adversarial way by prompting and by a classifier.”
Simple and Effective Retrieve-Edit-Rerank Text Generation (2020, background): “…text generation methods have received significant recent interest; editing humanauthored text can potentially avoid many of the challenges that are seen while generating text from scratch, including the tendency to be overly repetitive or to degrade on longer texts (Holtzman et al., 2018, 2019).”
GeDi: Generative Discriminator Guided Sequence Generation (2020, background): “…et al., 2021; Madotto et al., 2020; Keskar et al., 2019; Ziegler et al., 2019; Rajani et al., 2019; Fan et al., 2018; Ficler and Goldberg, 2017; Yu et al., 2017; Hu et al., 2017) or using a discriminator to guide decoding (Ghazvininejad et al., 2017; Holtzman et al., 2018; Dathathri et al., 2020).”
Plug and Play Language Models: A Simple Approach to Controlled Text Generation (2019, methodology): “Further, to avoid the problem of repetitive, low quality text (Holtzman et al., 2018), we compute the mean over the Dist-1, Dist-2 and Dist-3 scores (for the generated passage), which is an indicator of repetitiveness (Li et al.”
Towards Coherent and Cohesive Long-form Text Generation (2018, background): “Only recently have there been efforts to provide more relevant objectives, such as consistency and repetition in a text (Li et al., 2015, 2016a; Holtzman et al., 2018).”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, background): “NLPO accomplishes this via top-p sampling, which restricts tokens to the smallest possible set whose cumulative probability is greater than the probability parameter p (Holtzman et al., 2018).”
Neural Text Generation with Unlikelihood Training (2019, background): “Several forms of diverse beam search have been explored (Li et al., 2016; Vijayakumar et al., 2018; Kulikov et al., 2018; Holtzman et al., 2018) which can to some degree decrease the level of repetition of a model by selecting candidates unlike previously chosen ones, due to their variety.”
Generating Datasets with Pretrained Language Models (2021, methodology): “We apply top-p (Holtzman et al., 2020) and top-k (Fan et al., 2018; Holtzman et al., 2018) sampling with p = 0 .”
Recipes for Building an Open-Domain Chatbot (2020, background): “…models employing beam search decoding (or other methods that approximately choose the most likely utterance) tend to generate common words too frequently, and rare words too infrequently, as compared to the human distribution (Holtzman et al., 2018; Welleck et al., 2020; Li et al., 2019a).”
Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, background): “Class-conditional language models can also be used to learn disciminators to guide generation (Dathathri et al., 2019; Ghazvininejad et al., 2017; Holtzman et al., 2018; Krause et al., 2020).”
Neural Approaches to Conversational AI (2018, neutral): “as in GANs, but most trainable metrics in the literature do not exploit this iterative process. Adversarial setups proposed for dialogue and related tasks (Kannan and Vinyals, 2016; Li et al., 2017c; Holtzman et al., 2018) offer solutions to this problem, but it is also well-known that such setups s”
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts (2021, methodology): “Decoding-time methods, a more lightweight approach, have been used controlling the attributes of generated text, as well as for improving its quality (Li et al., 2016; Holtzman et al., 2018; Welleck et al., 2020).”
A Distributional Approach to Controlled Text Generation (2020, background): “As a proxy to perplexity, Holtzman et al. (2018) design hand-crafted rewards using a set of discriminators to ensure the quality of generated text in open-ended text generation.”
FUDGE: Controlled Text Generation With Future Discriminators (2021, methodology): “Our fine-tuning dataset was built by sampling directly from the original P (X) modeled by G to mitigate distribution shift, but it is well-known that language model generations are more repetitive than natural language (Holtzman et al., 2018, 2019).”
Content Planning for Neural Story Generation with Aristotelian Rescoring (2020, methodology): “Holtzman et al. (2018) generate continuation sentences from context sentences, and introduce using a mix of collaborative discriminators that each learn one Grician maxim of conversation, and use them to rerank their RNN story model output. GoldfarbTarrant et al. (2019) use those discriminators with”