Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control

By Natasha Jaques, Shixiang Shane Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E. Turner, Douglas Eck. 2016

222 citations recorded by the source corpus.

This paper proposes a general method for improving the structure and quality of sequences generated by a recurrent neural network (RNN), while maintaining information originally learned from data, as well as sample diversity. An RNN is first pre-trained on data using maximum likelihood estimation (MLE), and the probability distribution over the next token in the sequence learned by this model is treated as a prior policy. Another RNN is then trained using reinforcement learning (RL) to generate higher-quality outputs that account for domain-specific incentives while retaining proximity to the prior policy of the MLE RNN. To formalize this objective, we derive novel off-policy RL methods for RNNs from KL-control. The effectiveness of the approach is demonstrated on two applications; 1) generating novel musical melodies, and 2) computational molecular generation. For both problems, we show that the proposed method improves the desired properties and structure of the generated sequences, while maintaining information learned from data.

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Natural Language Processing, Alignment & RLHF, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Trevor Darrell
  3. Dacheng Tao
  4. Ming-Hsuan Yang
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Bernt Schiele

In Statistical and Nonlinear Physics

  1. Sergey Levine
  2. Chelsea Finn
  3. Jesse Engel
  4. Ilya Kostrikov
  5. Joelle Pineau
  6. Shixiang Shane Gu
  7. Jonathan Tompson
  8. Olivier Pietquin

In Natural Language Processing

  1. Yejin Choi
  2. Mohit Bansal
  3. Jianfeng Gao
  4. Percy Liang
  5. Hannaneh Hajishirzi
  6. Jason Aaron Edward Weston
  7. Luke S. Zettlemoyer
  8. Noah A. Smith

In Alignment & RLHF

  1. Ethan Perez
  2. Geoffrey Irving
  3. Angelica Chen
  4. Tomasz Korbak
  5. Yejin Choi
  6. Rafael Rafailov
  7. Stefano Ermon
  8. Sergey Levine

In Signal Processing

  1. Jesse Engel
  2. Adam Roberts
  3. Chelsea Finn
  4. Brooks Paige
  5. Stefano Ermon
  6. Natasha Jaques
  7. Jonathan Berant
  8. Alec Radford

Citations in the corpus