RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-Grained Correctional Human Feedback

By Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun and 1 more. Computer Vision and Pattern Recognition, 2023

466 citations recorded by the source corpus.

Multimodal Large Language Models (MLLMs) have recently demonstrated impressive capabilities in multimodal understanding, reasoning, and interaction. However, existing MLLMs prevalently suffer from serious hallucination problems, generating text that is not factually grounded in associated images. The problem makes existing MLLMs untrustworthy and thus impractical in real-world (especially high-stakes) applications. To address the challenge, we present RLHF-V, which enhances MLLM trustworthiness via behavior alignment from fine-grained correctional human feedback. Specifically, RLHF-V collects human preference in the form of segment-level corrections on hallucinations, and performs dense direct preference optimization over the human feedback. Comprehensive experiments on five benchmarks in both automatic and human evaluation show that, RLHF-V can enable substantially more trustworthy MLLM behaviors with promising data and computation efficiency. Remarkably, using 1.4k annotated data samples, RLHF-V significantly reduces the hallucination rate of the base MLLM by 34.8%, outperforming the concurrent LLaVA-RLHF trained on 10k annotated data. The final model achieves state-of-the-art performance in trustwor-thiness among open-source MLLMs, and shows better ro-bustness than GPT-4V in preventing hallucinations aroused from over-generalization.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0f9a3c5c6a54fca6be2afa0fd5fd34eed96a31e8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, background): “Yet, recent advancements in LLMs [32, 97, 103, 147] and some in MLLMs [132, 137] suggest that reinforcement learning from human or environmental feedback can further improve models, potentially surpassing the limits of supervised fine-tuning, especially in decision-making abilities.”
Hallucination of Multimodal Large Language Models: A Survey (2024, unknown): “Similarly, RLHF-V [119] also employs the RLHF paradigm to enhance the pre-trained MLLM. Specifically, this work emphasizes two improvements: 1) at the data level, it proposes to collect human feedback in the form of fine-grained segment-level corrections, providing a clear, dense, and fine-grained…”
A survey on multimodal large language models (2023, unknown): “…and previous reasoning history [14,65,75,76,82,83] Training-free Few-shot Chameleon [75], HuggingGPT [80], MM-REACT [14], VISPROG [76] Zero-shot CAT [84], CaFo [90], Visual ChatGPT [83], IdealGPT [91], ChatCaptioner [92], PointCLIP V2 [93], SMs [94] Finetuning GPT4Tools [34] Functions (§3.4.3)”
Visual-RFT: Visual Reinforcement Fine-Tuning (2025, unknown): “For LVLMs, RL has primarily been used for tasks like mitigating hallucinations and aligning models with human preference [21, 36, 37, 46, 47, 55, 60, 61], but there remains a significant gap in research focusing on enhancing reasoning and visual perception of Large Vision Language Models.”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, methodology): “In the domain of LLMs, the method of Reinforcement Learning with Human Feedback (RLHF) [138, 139] has been widely applied for model alignment.”