Uncertainty Estimation for Language Reward Models

By Adam Gleave, Geoffrey Irving. 2022

41 citations recorded by the source corpus.

Language models can learn a range of capabilities from unsupervised training on text corpora. However, to solve a particular problem (such as text summarization) it is typically necessary to fine-tune them on a task-specific dataset. It is often easier for humans to choose between options than to provide labeled data, and prior work has achieved state-of-the-art performance by training a reward model from such preference comparisons. However, collecting a large preference comparison dataset is still expensive -- and the learned reward models are unreliable out-of-distribution. We seek to address these problems via uncertainty estimation, which can improve sample efficiency and robustness using active learning and risk-averse reinforcement learning (RL). Specifically, we use bootstrap aggregating (bagging) to train an ensemble of reward models differing in the initialization of their final layer. Ensembles have proved successful in prior applications of active learning, but we find that in our setting ensemble active learning does not outperform random sampling. Further experiments show that while the aggregate predictions are well-calibrated, the ensemble's estimated epistemic uncertainty is only weakly correlated with model error. We suspect this is because the ensemble members are fine-tuned from a single model and so are similar to one another. This suggests current pre-training methods will need to be modified to support uncertainty estimation, e.g. by training multiple language models.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Scaling Laws for Reward Model Overoptimization (2022, neutral): “Many approaches have been proposed for reducing overoptimization in general [Taylor, 2016, Everitt et al., 2017], as well as in RMs [Gleave and Irving, 2022], including within the ﬁeld of adversarial robustness [Chakraborty et al., 2018].”
Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback (2023, background): “, 2017), as well as in reward models (Gleave and Irving, 2022).”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “However, active learning techniques in deep learning rely on heuristics for prediction confidence which can be unreliable [Gleave and Irving, 2022].”