Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback
By Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Qiuyuan Huang, Lars Lidén, Zhou Yu, Weizhu Chen, Jianfeng Gao. 2023
530 citations recorded by the source corpus.
Large language models (LLMs), such as ChatGPT, are able to generate human-like, fluent responses for many downstream tasks, e.g., task-oriented dialog and question answering. However, applying LLMs to real-world, mission-critical applications remains challenging mainly due to their tendency to generate hallucinations and their inability to use external knowledge. This paper proposes a LLM-Augmenter system, which augments a black-box LLM with a set of plug-and-play modules. Our system makes the LLM generate responses grounded in external knowledge, e.g., stored in task-specific databases. It also iteratively revises LLM prompts to improve model responses using feedback generated by utility functions, e.g., the factuality score of a LLM-generated response. The effectiveness of LLM-Augmenter is empirically validated on two types of scenarios, task-oriented dialog and open-domain question answering. LLM-Augmenter significantly reduces ChatGPT's hallucinations without sacrificing the fluency and informativeness of its responses. We make the source code and models publicly available.
Subfields: Artificial Intelligence, Health Informatics, Retrieval-Augmented Generation, Large Language Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Percy Liang
- Luke S. Zettlemoyer
- Yejin Choi
- Noah A. Smith
- Graham Neubig
- Quoc V. Le
In Health Informatics
- Amit P. Sheth
- Amitava Das
- Vipula Rawte
- Alexander J. Ratner
- Noah A. Smith
- Song-Chun Zhu
- Niket Tandon
- Sarah Wiegreffe
In Retrieval-Augmented Generation
- Luyu Gao
- Ji-rong Wen
- Wayne Xin Zhao
- Jianfeng Gao
- Michel Galley
- Mojtaba Komeili
- Noah A. Smith
- Ranjay Krishna
In Large Language Models
Citations in the corpus
- Self-Refine: Iterative Refinement with Self-Feedback (2023, methodology): “Prompted refiners: Augmenter (Peng et al., 2023), Re(3) (Yang et al.”
- Instruction Tuning with GPT-4 (2023, background): “Meanwhile, recent studies show that GPT-4 is capable of identifying and fixing its own mistakes, and accurately judging the quality of responses(Peng et al., 2023; Bai et al., 2022; Madaan et al., 2023; Kim et al., 2023).”
- The rise and potential of large language model based agents: a survey (2025, unknown): “Additionally, integrating external knowledge bases and databases can mitigate hallucination issues [84, 547].”
- Demystifying GPT Self-Repair for Code Generation (2023, background): “[29] use self-repair to mitigate hallucinations and improve factual grounding in a ChatGPT-based web search assistant, in which the model revises its initial response based on self-generated feedback.”
- ToolQA: A Dataset for LLM Question Answering with External Tools (2023, background): “For example, a Wolfram math plugin can enhance numerical reasoning [68], and a verified database can mitigate hallucinations by providing up-to-date fact-checked knowledge [49].”
- How Language Model Hallucinations Can Snowball (2023, background): “Hallucination is commonly attributed to knowledge gaps in LMs (Zheng et al., 2023), motivating mitigation strategies through retrieval over knowledge bases (Lewis et al., 2020; Shuster et al., 2021; Peng et al., 2023) But, do LMs only hallucinate when they do not “know” a fact?”
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (2023, background): “The latest works study how to utilize LLMs interacting with tools to expand their knowledge as a plugin, including extending LLM to process multimodality content [64, 51], teaching LLMs to access the internet with correct API calls [49], and expanding their knowledge with local databases to accompli”
- CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, background): “, 2022) SL & ICL Trained reranker Scalar ✓ (post-hoc) High ✗ LLM-Augmenter (Peng et al., 2023) RL External Knowledge NL ✓ (post-hoc) High ✗ CAI(Bai et al.”
- Large Language Models: A Survey (2024, unknown): “LLMs can also be augmented by using external knowledge and tools [35], [36] so that they can effectively interact with users and environment [37], and continually improve itself using feedback data collected through interactions (e.g. via reinforcement learning with human feedback (RLHF)).”
- Chain-of-Verification Reduces Hallucination in Large Language Models (2023, methodology): “Other approaches include using tools for fact-checking (Chern et al., 2023a; Galitsky, 2023; Peng et al., 2023), or linking to external documents with attribution (Menick et al.”
- StructGPT: A General Framework for Large Language Model to Reason over Structured Data (2023, background): “, 2023), and also fall short of mastering domain-specific or real-time knowledge (Schick et al., 2023; Peng et al., 2023; Jiang et al., 2022a).”
- The Internal State of an LLM Knows When its Lying (2023, methodology): “Some methods for reducing hallucination assume that the LLM is a black box (Peng et al., 2023).”
- Hallucination is Inevitable: An Innate Limitation of Large Language Models (2024, background): “Similar to programmable guardrails and fences, it explicitly controls the LLM workflow by changing how information is recalled (through retrieval from knowledge database [39, 48]) or used (through logic inference, which has been done for deep networks as in [71, 72]).”
- Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (2023, background): “Another active research area is to incorporate external verification [39, 32] and human feedback [35] to allow LLMs to perform more reliably and align better with human preferences.”
- Benchmarking Large Language Models in Retrieval-Augmented Generation (2023, background): “2021; Trivedi et al. 2023; Li et al. 2023a), dialogue (Cai et al. 2019a,b; Peng et al. 2023), domain-specific question answering (Cui et al. 2023) and code generation (Zhou et al. 2023b).”