Fine-Grained Human Feedback Gives Better Rewards for Language Model Training

By Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, Hannaneh Hajishirzi. Neural Information Processing Systems, 2023

481 citations recorded by the source corpus.

Language models (LMs) often exhibit undesirable text generation behaviors, including generating false, toxic, or irrelevant outputs. Reinforcement learning from human feedback (RLHF) - where human preference judgments on LM outputs are transformed into a learning signal - has recently shown promise in addressing these issues. However, such holistic feedback conveys limited information on long text outputs; it does not indicate which aspects of the outputs influenced user preference; e.g., which parts contain what type(s) of errors. In this paper, we use fine-grained human feedback (e.g., which sentence is false, which sub-sentence is irrelevant) as an explicit training signal. We introduce Fine-Grained RLHF, a framework that enables training and learning from reward functions that are fine-grained in two respects: (1) density, providing a reward after every segment (e.g., a sentence) is generated; and (2) incorporating multiple reward models associated with different feedback types (e.g., factual incorrectness, irrelevance, and information incompleteness). We conduct experiments on detoxification and long-form question answering to illustrate how learning with such reward functions leads to improved performance, supported by both automatic and human evaluation. Additionally, we show that LM behaviors can be customized using different combinations of fine-grained reward models. We release all data, collected human feedback, and codes at https://FineGrainedRLHF.github.io.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/e2e52461194bc81351da7caa978ac42e9e9549cc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback (2023, unknown): “As a result, existing preference datasets are either small in scale (Wu et al., 2023) or limited on specific tasks (Stiennon et al., 2020; Nakano et al., 2021).”
WildChat: 1M ChatGPT Interaction Logs in the Wild (2024, methodology): “…behavior with human expectations, and (3) optionally applying Reinforcement Learning from Human Feedback (RLHF) to further optimize the model’s responses based on human preferences (Stiennon et al., 2020; Ouyang et al., 2022; Rama-murthy et al., 2023; Wu et al., 2023; Rafailov et al., 2023).”
Detecting and Preventing Hallucinations in Large Vision Language Models (2023, methodology): “Similar to [28], this should give stronger signal during training in reducing hallucinatory generations as compared to using pairs of likelihoods over entire generations.”
RewardBench: Evaluating Reward Models for Language Modeling (2024, background): “RLHF for alignment has been operationalized beyond general preference by comparing aspect-based preference, where aspects could be more general concepts like helpfulness or harmlessness (Bai et al., 2022a), or more fine-grained ones (Wu et al., 2023), among others.”
Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (2023, background): “Compared to ORM, PRM provides more detailed feedback, demonstrating greater potential to enhance generator (Wu et al., 2023).”
UltraFeedback: Boosting Language Models with High-quality Feedback (2023, unknown): “Some datasets are small in scale (Wu et al., 2023; Wang et al., 2023c) or provide only community votes as coarse-grained preferences (Ethayarajh et al., 2022; Askell et al., 2021).”
OpenChat: Advancing Open-source Language Models with Mixed-Quality Data (2023, methodology): “Another intuitive approach to align LLMs is through RL, which models rewards according to human preference feedbacks (Ouyang et al., 2022; Bai et al., 2022a; Rafailov et al., 2023) or pre-defined classifiers (Wu et al., 2023), and fine-tune LLMs to maximize the reward.”
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (2023, background): “Manual annotation of reflection tokens for each segment is expensive (Wu et al., 2023).”
Safe RLHF: Safe Reinforcement Learning from Human Feedback (2023, unknown): “Based on these models, LLMs are guided to be fine-tuned to ensure that the models integrate multiple properties (Glaese et al., 2022; Wu et al., 2023b; Touvron et al., 2023a).”
RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-Grained Correctional Human Feedback (2023, background): “In comparison, RLHF-V presents the first fine-grained correctional human feedback learning framework for behavior alignment, and systematically addresses different hallucination sources in training MLLMs, achieving strong performance in trustworthiness.”
A Long Way to Go: Investigating Length Correlations in RLHF (2023, methodology): “The last half of training usually yields a decrease in KL divergence, as the model has optimized for reward and is regularized closer to the initial policy model by the KL term.”
Evaluating Large Language Models at Evaluating Instruction Following (2023, methodology): “In the context of instruction tuning, to replace the costly and unreproducible human evaluation (Ouyang et al., 2022; Zhao et al., 2023; Wu et al., 2023a), many recent works take prompted LLMs as evaluators to compare model outputs (Chiang et al., 2023; Peng et al., 2023; Dubois et al., 2023; Zhou…”
Making Large Language Models Better Reasoners with Alignment (2023, unknown): “There are three primary categories of preference alignment methods: 1) Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022; Wu et al., 2023), which trains a reward model by utilizing human feedback.”
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, unknown): “Strategies such as training reward models help discriminate between desirable and undesirable outputs (Lightman et al., 2023; Wu et al., 2023; Chen et al., 2024b).”
BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset (2023, methodology): “Safe-RLHF : Our proposed method leverages both cost and reward models and is trained using the PPO-Lagrangian [59, 60] algorithm.”
Chain-of-Verification Reduces Hallucination in Large Language Models (2023, methodology): “This includes using re-inforcement learning (Roit et al., 2023; Wu et al., 2023), constrastive learning (Chern et al., 2023b; Sun et al., 2023b) and other methods (Li et al., 2023; Narayan et al., 2023).”
Preference Ranking Optimization for Human Alignment (2023, background): “complex than supervised learning, prone to optimization instability, and sensitive to hyperparameters (Rafailov et al., 2023; Wu et al., 2023; Yuan et al., 2023).”