On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting

By Tomasz Korbak, Hady ElSahar, Germán Kruszewski, Marc Dymetman. 2022

94 citations recorded by the source corpus.

The availability of large pre-trained models is changing the landscape of Machine Learning research and practice, moving from a training-from-scratch to a fine-tuning paradigm. While in some applications the goal is to"nudge"the pre-trained distribution towards preferred outputs, in others it is to steer it towards a different distribution over the sample space. Two main paradigms have emerged to tackle this challenge: Reward Maximization (RM) and, more recently, Distribution Matching (DM). RM applies standard Reinforcement Learning (RL) techniques, such as Policy Gradients, to gradually increase the reward signal. DM prescribes to first make explicit the target distribution that the model is fine-tuned to approximate. Here we explore the theoretical connections between the two paradigms, and show that methods such as KL-control developed for RM can also be construed as belonging to DM. We further observe that while DM differs from RM, it can suffer from similar training difficulties, such as high gradient variance. We leverage connections between the two paradigms to import the concept of baseline into DM methods. We empirically validate the benefits of adding a baseline on an array of controllable language generation tasks such as constraining topic, sentiment, and gender distributions in texts sampled from a language model. We observe superior performance in terms of constraint satisfaction, stability and sample efficiency.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Alignment & RLHF

Citations in the corpus

Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “⇣ ⌘ Even if we use the MLE estimate r � of the ground-truth reward function r ⇤ , it is still expensive to estimate the partition function Z ( x ) [17, 15], which makes this representation hard to utilize in practice. is the partition function.”
Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “8 is very similar, with a “pseudo-reward” rθ(x) = −f ′ (πθ(x)p(x) ), one difference being that now rθ depends on θ (see (Korbak et al., 2022b) for related remarks).”
RL with KL penalties is better viewed as Bayesian inference (2022, background): “However, J GDC ( θ ) is no longer equivalent to RL (Korbak et al., 2022b) because the expectation in forward KL divergence is with respect to a π ∗ KL-RL , not π θ .”
Class-Incremental Learning: A Survey (2023, background): “…[137], [242], [243], [244], [245], [246], [247], video understanding [91], [161], medical surgery [248], [249], language model [250], [251], [252], [253], language-vision model [217], [254], etc. Specifically, object detection and semantic segmentation are two popular vision tasks that may also…”
Deep Class-Incremental Learning: A Survey (2023, unknown): “, semantic segmentation [164], [165], [166], [167], [174], object detection [175], [176], [177], [178], [179], [180], [181], video understanding [95], [133], medical surgery [182], [183], language model [184], [185], [186], [187], language-vision model [188], [189], etc.”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “This perspective on RLHF separates the modeling problem (defining a target distribution specifying the desired behavior of an LLM) and the inference problem (approximating that target distribution) [Korbak et al., 2022a, Go et al., 2023].”