Text Generation by Learning from Demonstrations

By Richard Yuanzhe Pang, He He. 2020

88 citations recorded by the source corpus.

Current approaches to text generation largely rely on autoregressive models and maximum likelihood estimation. This paradigm leads to (i) diverse but low-quality samples due to mismatched learning objective and evaluation metric (likelihood vs. quality) and (ii) exposure bias due to mismatched history distributions (gold vs. model-generated). To alleviate these problems, we frame text generation as an offline reinforcement learning (RL) problem with expert demonstrations (i.e., the reference), where the goal is to maximize quality given model-generated histories. We propose GOLD (generation by off-policy learning from demonstrations): an easy-to-optimize algorithm that learns from the demonstrations by importance weighting. Intuitively, GOLD upweights confident tokens and downweights unconfident ones in the reference during training, avoiding optimization issues faced by prior RL approaches that rely on online data collection. According to both automatic and human evaluation, models trained by GOLD outperform those trained by MLE and policy gradient on summarization, question generation, and machine translation. Further, our models are less sensitive to decoding algorithms and alleviate exposure bias.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Graham Neubig
3. Mohit Bansal
4. Zhe Gan
5. Percy Liang
6. Eric P. Xing
7. Yoshua Bengio
8. Pengfei Liu

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Jianfeng Gao
3. Dacheng Tao
4. Trevor Darrell
5. Cordelia Schmid
6. Xiaogang Wang
7. Zhe Gan
8. Ming-Hsuan Yang

In Reinforcement Learning

1. T. Paine
2. Caglar Gulcehre
3. Dragomir R. Radev
4. Pengfei Liu
5. Yixin Liu
6. Graham Neubig
7. Wanqiao Xu
8. Jesse Mu

In Natural Language Generation

1. Pengfei Liu
2. Dragomir R. Radev
3. Yixin Liu
4. Graham Neubig
5. Wanqiao Xu
6. Jesse Mu
7. G. Qian
8. P. Dupenloup

Citations in the corpus

• Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, neutral): “…et al., 2020; Paulus et al., 2017), dialogue (Li et al., 2016; Zhou et al., 2017; Jaques et al., 2020), image captioning (Rennie et al., 2017), question generation (Pang & He, 2021), text-games (Narasimhan et al., 2015; Hausknecht et al., 2020), and more (Ranzato et al., 2016; Snell et al., 2022).”
• AgreeSum: Agreement-Oriented Multi-Document Summarization (2021, background): “It is timely to revisit the problem also because recent stronger neural abstractive summarization models are prone to hallucination, as are neural text generation in general (Wiseman et al., 2017; Tian et al., 2019; Wang and Sennrich, 2020; Pang and He, 2020).”
• Text Generation with Efficient (Soft) Q-Learning (2021, methodology): “We compare our training approach with several baselines, including (1) the standard MLE training ( MLE ), (2) PG with MLE initialization ( MLE+PG ), and (3) one of the latest text-generation RL algorithms GOLD-s [42] which is a pure off-policy method based on importance-sampling PG.”
• Reinforced Self-Training (ReST) for Language Modeling (2023, methodology): “The GOLD loss was introduced by Pang and He (2021).”
• BRIO: Bringing Order to Abstractive Summarization (2022, methodology): “GOLD (Pang and He, 2021) uses offline reinforcement learning to train the BART model by treating the reference summaries as the demonstrations, a different formulation that can also improve the performance of the original BART.”
• A Comprehensive Overview of Large Language Models (2023, unknown): “The pre-training employs standard language modeling objectives, while GOLD [122] with tempering [123] serves as the training objective for the fine-tuning on CodeContests data.”
• Self-Edit: Fault-Aware Code Editor for Code Generation (2023, methodology): “address this discrepancy, we follow previous work and adopt a variation of GOLD (Pang and He, 2021; Li et al., 2022c), which incorporates an off-policy importance weight into the standard maximum likelihood objective gradient:”
• Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone (2022, methodology): “We find that integrating GOLD [48] into F IBER can bring significant improvements, outperforming models trained with hundreds of millions of images.”
• Reinforcement Learning for Language Models (2023, methodology): “In the NLP context, RL has been used to improve model behavior in many tasks such as automatic translation [5, 6], question generation [7], summarization [8, 9], dialogue generation [10, 11], and text-based games [12, 13], to name a few.”