Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control
By Natasha Jaques, Shixiang Shane Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E. Turner, Douglas Eck. 2016
222 citations recorded by the source corpus.
This paper proposes a general method for improving the structure and quality of sequences generated by a recurrent neural network (RNN), while maintaining information originally learned from data, as well as sample diversity. An RNN is first pre-trained on data using maximum likelihood estimation (MLE), and the probability distribution over the next token in the sequence learned by this model is treated as a prior policy. Another RNN is then trained using reinforcement learning (RL) to generate higher-quality outputs that account for domain-specific incentives while retaining proximity to the prior policy of the MLE RNN. To formalize this objective, we derive novel off-policy RL methods for RNNs from KL-control. The effectiveness of the approach is demonstrated on two applications; 1) generating novel musical melodies, and 2) computational molecular generation. For both problems, we show that the proposed method improves the desired properties and structure of the generated sequences, while maintaining information learned from data.
Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Natural Language Processing, Alignment & RLHF, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Trevor Darrell
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Bernt Schiele
In Statistical and Nonlinear Physics
- Sergey Levine
- Chelsea Finn
- Jesse Engel
- Ilya Kostrikov
- Joelle Pineau
- Shixiang Shane Gu
- Jonathan Tompson
- Olivier Pietquin
In Natural Language Processing
- Yejin Choi
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Hannaneh Hajishirzi
- Jason Aaron Edward Weston
- Luke S. Zettlemoyer
- Noah A. Smith
In Alignment & RLHF
- Ethan Perez
- Geoffrey Irving
- Angelica Chen
- Tomasz Korbak
- Yejin Choi
- Rafael Rafailov
- Stefano Ermon
- Sergey Levine
In Signal Processing
- Jesse Engel
- Adam Roberts
- Chelsea Finn
- Brooks Paige
- Stefano Ermon
- Natasha Jaques
- Jonathan Berant
- Alec Radford
Citations in the corpus
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “In particular, we formulate the following optimization problem max where � is a parameter controlling the deviation from the base reference policy ⇡ ref , namely the initial SFT model ⇡ SFT .”
- Fine-Tuning Language Models from Human Preferences (2019, methodology): “…such as BLEU for translation (Ranzato et al., 2015; Wu et al., 2016), ROUGE for summarization (Ranzato et al., 2015; Paulus et al., 2017; Wu and Hu, 2018; Gao et al., 2019b), music theory-based rewards (Jaques et al., 2017), or event detectors for story generation (Tambwekar et al., 2018).”
- Learning to summarize from human feedback (2020, background): “A human judges which is a better summary of the post.”
- Red Teaming Language Models with Language Models (2022, methodology): “To prevent RL from collapsing to a single, highreward generation, we add a loss term to penalize KL divergence between pr(x) and initialization’s distribution over next tokens (Jaques et al., 2017; Schmitt et al., 2018; Jaques et al., 2019; Ziegler et al., 2019).”
- On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, methodology): “These preferences could include topic and sentiment (Dathathri et al., 2020), valid musical notes and molecular structures (Jaques et al., 2017a), code compilability (Korbak et al., 2021), reducing distributional biases (Khalifa et al., 2021; Weidinger et al., 2021), evaluation metrics for Machine…”
- Human-centric Dialog Training via Offline Reinforcement Learning (2020, methodology): “KL-control has been used to improve transfer learning between maximum likelihood estimation (MLE) training on data, and training with RL (Jaques et al., 2017).”
- Teaching language models to support answers with verified quotes (2022, methodology): “To prevent RL from collapsing to a single, high-reward generation, we add a loss term to penalize KL divergence between 𝑝𝑟 (𝑥) and initialization’s distribution over next tokens (Jaques et al., 2017, 2019; Schmitt et al., 2018; Ziegler et al., 2019).”
- Opportunities and obstacles for deep learning in biology and medicine (2017, background): “The initial RNN is then fine-tuned to generate molecules that are likely to be active against a specific target by either continuing training on a small set of positive examples [323] or adopting reinforcement learning strategies [326,327].”
- Grammar Variational Autoencoder (2017, background): “Generative machine learning models have been used recently to produce extraordinary results, from realistic musical improvisation (Jaques et al., 2016), to changing facial expressions in images (Radford et al., 2015; Upchurch et al., 2016), to creating realistic looking artwork (Gatys et al., 2015).”
- Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning (2020, background): “…on policy priors that are optimized to be different from the actual policy but so far mainly in the multi-task or transfer-learning setup, i.e. to share knowledge across or to transfer knowledge to new tasks (Teh et al., 2017; Galashov et al., 2019; Tirumala et al., 2019; Jaques et al., 2017).”
- Offline Reinforcement Learning with Fisher Divergence Critic Regularization (2021, methodology): “, 2019), reminiscent of KL-control methods in related literature (Kappen et al., 2012; Jaques et al., 2017).”
- EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, neutral): “[19], Haarnoja et al.”
- Deep Communicating Agents for Abstractive Summarization (2018, background): “Reinforcement Learning (RL) Loss Policy gradient methods can directly optimize discrete target evaluation metrics such as ROUGE that are non-differentiable (Paulus et al., 2018; Jaques et al., 2017; Pasunuru and Bansal, 2017; Wu et al., 2016).”
- A Distributional Approach to Controlled Text Generation (2020, background): “Jaques et al. (2017); Ziegler et al. (2019) propose a conservative fine-tuning approach moderated by a KL penalty between the trained policy and the original LM, discouraging large deviations.”
- Benchmarking Batch Deep Reinforcement Learning Algorithms (2019, methodology): “Rather than a hard maximum in the target, Ψ-learning uses the log over the sum of the exponential of the action-values [Jaques et al., 2017].”
- Explaining Question Answering Models through Text Generation (2020, methodology): “We add a KLD loss regularization term to the generator objective (Jaques et al., 2016): given the original pre-trained LM p NL , we minimize at each decoding step the KL divergence D ( p gen || p NL ) between the trained generator distribution and the original pre-trained LM distribution.”
- Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Models (2017, background): “For example, [Jaques et al., 2016] introduce penalties to unrealistic sequences, in absence of which RL can easily get stuck around local maxima which can be very far from the global maximum reward.”