Aligning Large Multimodal Models with Factually Augmented RLHF

By Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang and 2 more. Annual Meeting of the Association for Computational Linguistics, 2023

811 citations recorded by the source corpus.

Large Multimodal Models (LMM) are built across modalities and the misalignment between two modalities can result in"hallucination", generating textual outputs that are not grounded by the multimodal information in context. To address the multimodal misalignment issue, we adapt the Reinforcement Learning from Human Feedback (RLHF) from the text domain to the task of vision-language alignment, where human annotators are asked to compare two responses and pinpoint the more hallucinated one, and the vision-language model is trained to maximize the simulated human rewards. We propose a new alignment algorithm called Factually Augmented RLHF that augments the reward model with additional factual information such as image captions and ground-truth multi-choice options, which alleviates the reward hacking phenomenon in RLHF and further improves the performance. We also enhance the GPT-4-generated training data (for vision instruction tuning) with previously available human-written image-text pairs to improve the general capabilities of our model. To evaluate the proposed approach in real-world scenarios, we develop a new evaluation benchmark MMHAL-BENCH with a special focus on penalizing hallucinations. As the first LMM trained with RLHF, our approach achieves remarkable improvement on the LLaVA-Bench dataset with the 94% performance level of the text-only GPT-4 (while previous best methods can only achieve the 87% level), and an improvement by 60% on MMHAL-BENCH over other baselines. We opensource our code, model, data at https://llava-rlhf.github.io.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/844bb298d49ef4a07b5d4929dfdfd170f6a1d5f5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

VILA: On Pre-training for Visual Language Models (2023, neutral): “Most of the efforts in this field have been focusing on improving the visual language instruction-tuning process, i.e ., supervised fine-tuning (SFT) or reinforcement learning from human feedback (RLHF) [36, 37, 52].”
Visual-RFT: Visual Reinforcement Fine-Tuning (2025, neutral): “For LVLMs, RL has primarily been used for tasks like mitigating hallucinations and aligning models with human preference [21, 36, 37, 46, 47, 55, 60, 61], but there remains a significant gap in research focusing on enhancing reasoning and visual perception of Large Vision Language Models.”
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding (2023, background): “Several recent studies address this issue by proposing hallucination-targeted datasets for fine-tuning [20, 42], training a post-hoc revisor to reconstruct less hallucinatory outputs [77] or adapting factually augmented Reinforcement Learning from Human Feedback (RLHF) [60].”
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2023, methodology): “Additionally, the computational and economic costs of our system include multiple LVLM (LLaVA [56]) and one or more proprietary LLM inference(s) when building and querying the scenegraph, which may be significant.”
Hallucination of Multimodal Large Language Models: A Survey (2024, neutral): “Insufficient data could potentially lead to problematic cross-modal alignment, resulting in hallucinations [96, 103].”
RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-Grained Correctional Human Feedback (2023, methodology): “Despite the success, current MLLMs suffer from serious hallucination problems [21, 24, 25, 37].”
OtterHD: A High-Resolution Multi-modality Model (2023, methodology): “…Mixture We compiled a total of 370K instruction/response pairs sourced from the following public datasets: LLaVA-Instruct [30], VQAv2 [2], GQA [23], OKVQA [36], OCRVQA [38], A-OKVQA [45], COCO-GOI [33], COCO-Caption [10], TextQA [48], RefCOCO [58], COCO-ITM [28], ImageNet [17], and LLaVA-RLHF [51].”
A survey on multimodal large language models (2023, neutral): “…and previous reasoning history [14,65,75,76,82,83] Training-free Few-shot Chameleon [75], HuggingGPT [80], MM-REACT [14], VISPROG [76] Zero-shot CAT [84], CaFo [90], Visual ChatGPT [83], IdealGPT [91], ChatCaptioner [92], PointCLIP V2 [93], SMs [94] Finetuning GPT4Tools [34] Functions (§3.4.3)”
A Survey on Hallucination in Large Vision-Language Models (2024, background): “Consequently, misalignment could be a key factor in hallucination generation [Sun et al. , 2023; Zhao et al. , 2023b].”
InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, neutral): “We evaluate InternVL’s propensity for hallucinations on four established benchmarks—HallusionBench [45], MMHal-Bench [111], CRPE [126], and POPE [67]—as detailed in Table 5.”
What matters when building vision-language models? (2024, neutral): “Although EVA-CLIP-5B (Sun et al., 2023) is ten times bigger in parameter counts than SigLIP-SO400M (Zhai et al., 2023), we obtain similar performance across 4 benchmarks, suggesting that EVA-CLIP-5B could be heavily under-trained, and we acknowledge that the open VLM community is missing a large wel”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “We evaluate the performance of InternVL on four key hallucination evaluation benchmarks: HallusionBench [77], MMHal [223], CRPE [250], and POPE [139].”
GPT-4V(ision) is a Generalist Web Agent, if Grounded (2024, methodology): “MindAct (Deng et al., 2023) instead employs a small language model to rank each HTML element and selectively consider top elements as the context.”
SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training (2025, neutral): “…(SFT) (Radford et al., 2018; Brown et al., 2020; Radford et al., 2021; Wei et al., 2022a; Chung et al., 2022; Zhou et al., 2024a) and/or reinforcement learning (RL) (Ziegler et al., 2019; Ouyang et al., 2022; Sun et al., 2024; Abdulhai et al., 2023; Zhou et al., 2024b; Zhai et al., 2024a).”