Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning

By Louis Castricato, Alex Havrilla, Shahbuland Matiana, M. Pieler, Anbang Ye, I. Yang, Spencer J. Frazier, Mark O. Riedl. 2022

15 citations recorded by the source corpus.

Controlled automated story generation seeks to generate natural language stories satisfying constraints from natural language critiques or preferences. Existing methods to control for story preference utilize prompt engineering which is labor intensive and often inconsistent. They may also use logit-manipulation methods which require annotated datasets to exist for the desired attributes. To address these issues, we first train a contrastive bi-encoder model to align stories with corresponding human critiques, named CARP, building a general purpose preference model. This is subsequently used as a reward function to fine-tune a generative language model via reinforcement learning. However, simply fine-tuning a generative language model with a contrastive reward model does not always reliably result in a story generation system capable of generating stories that meet user preferences. To increase story generation robustness we further fine-tune the contrastive reward model using a prompt-learning technique. A human participant study is then conducted comparing generations from our full system, ablations, and two baselines. We show that the full fine-tuning pipeline results in a story generator preferred over a LLM 20x as large as well as logit-based methods. This motivates the use of contrastive learning for general purpose human preference modeling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Roberta Raileanu
2. Jelena Luketina
3. Edward Grefenstette
4. Eric Hambro
5. Christoforos Nalmpantis
6. Mohit Bansal
7. Phil Blunsom
8. Yoav Goldberg

In Computer Vision and Pattern Recognition

1. Jelena Luketina
2. Edward Grefenstette
3. Christoforos Nalmpantis
4. Roberta Raileanu
5. Mohit Bansal

In Reinforcement Learning

1. Jelena Luketina
2. Edward Grefenstette
3. Christoforos Nalmpantis
4. Roberta Raileanu

Citations in the corpus

• GPTScore: Evaluate as You Desire (2023, neutral): “Unlike previous works (Matiana et al., 2021; Xu et al., 2022a,b; Castricato et al., 2022) that only consider the overall text quality, we focus on evaluating multi-dimensional text quality.”
• Understanding the Effects of RLHF on LLM Generalisation and Diversity (2023, methodology): “However, in use cases where the model needs to generate a wide variety of outputs, such as story generation (Castricato et al., 2022), red-teaming (Perez et al.”