Factuality Enhanced Language Models for Open-Ended Text Generation

By Nayeon Lee, Wei Ping, Peng Xu, M. Patwary, Mohammad Shoeybi, Bryan Catanzaro. 2022

310 citations recorded by the source corpus.

Pretrained language models (LMs) are susceptible to generate text with nonfactual information. In this work, we measure and improve the factual accuracy of large-scale LMs for open-ended text generation. We design the FactualityPrompts test set and metrics to measure the factuality of LM generations. Based on that, we study the factual accuracy of LMs with parameter sizes ranging from 126M to 530B. Interestingly, we find that larger LMs are more factual than smaller ones, although a previous study suggests that larger LMs can be less truthful in terms of misconceptions. In addition, popular sampling algorithms (e.g., top-p) in open-ended text generation can harm the factuality due to the ''uniform randomness'' introduced at every sampling step. We propose the factual-nucleus sampling algorithm that dynamically adapts the randomness to improve the factuality of generation while maintaining quality. Furthermore, we analyze the inefficiencies of the standard training method in learning correct associations between entities from factual text corpus (e.g., Wikipedia). We propose a factuality-enhanced training method that uses TopicPrefix for better awareness of facts and sentence completion as the training objective, which can vastly reduce the factual errors. We release our code and FactualityPrompts benchmark at: https://github.com/nayeon7lee/FactualityPrompt.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Generative Models

Citations in the corpus

BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “In circumstances where beam search is notably useful (i.e., dialogue generation), we instead employ the recently introduced factual nucleus sampling (Lee et al., 2022); we found this method to provide an appropriate balance between diversity of downstream generation while avoiding the hallucinatory…”
Survey of Hallucination in Natural Language Generation (2022, background): “[73] show that deliberately added łrandomnessž from sampling-based decoding increases the unexpected nature of the generation and the higher chance of containing hallucinated content.”
“According to . . . ”: Prompting Language Models Improves Quoting from Pre-Training Data (2023, methodology): “…(Sun et al., 2023; Borgeaud et al., 2022; Mallen et al., 2023; Weller et al., 2022), applying new decoding approaches (He et al., 2022), post hoc tuning of LLMs (Menick et al., 2022; Lee et al., 2022b), and analyzing the model’s output training data (Han and Tsvetkov, 2022; Park et al., 2023).”
OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation (2023, unknown): “Researches on mitigating current LLMs’ hallucination issues often focuses on several aspects, including refining the training process, us-ing larger and more diverse datasets [24], or implementing post-training evaluation [11] and correction mechanisms [33, 34].”
Improving Factuality and Reasoning in Language Models through Multiagent Debate (2023, methodology): “To improve factuality, approaches have relied on training techniques such as RLHF [33, 16, 2], pruning truthful datasets [12], external knowledge retrieval [7] and training-free methods based off likelihood estimation [10].”
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023, background): “Lee et al. (2022) evaluates the factual precision with proxy metrics, e.g., whether named entities in a generation appear in an article of the topic.”
Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models (2022, background): “[26] Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro.”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, unknown): “Thirdly, a line of work (Lee et al., 2022; Muhlgay et al., 2023) Table 3: Representative benchmarks that can be used for evaluating LLM fact-conflicting hallucination.”
On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024, unknown): “Chain-of-Thought (CoT) prompting stands out as one of the most popular strategies for improving the faithfulness of LLM-generated content (Wei et al., 2022).”
A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity (2023, background): “LLMs are known to be susceptible to generating nonfactual, untruthful information, which is referred to as hallucination (Lee et al., 2022; Ji et al., 2022a,b; Su et al., 2022; Dai et al., 2023b; Xu et al., 2023).”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, background): “, 2021) is a critical issue for LLMs because it may hallucinate incorrect output that is hard to distinguish (Lin et al., 2022b; Lee et al., 2022), especially when relying on parametric memory (Lewis et al.”
Hallucination is Inevitable: An Innate Limitation of Large Language Models (2024, methodology): “Factuality-enhanced training objectives [32, 56] have been shown to improve the model’s ability to associate facts and produce logical outputs.”
Knowledge Conflicts for LLMs: A Survey (2024, unknown): “However, this randomness can cause intra-memory conflicts, where the model produces different outputs for the same input due to the left-to-right generation pattern and the influence of sampled tokens on subsequent generations (Lee et al., 2022b; Huang et al., 2023; Dziri et al., 2021).”