Entity-Based Knowledge Conflicts in Question Answering

By Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, Sameer Singh. 2021

396 citations recorded by the source corpus.

Knowledge-dependent tasks typically use two sources of knowledge: parametric, learned at training time, and contextual, given as a passage at inference time. To understand how models use these sources together, we formalize the problem of knowledge conflicts, where the contextual information contradicts the learned information. Analyzing the behaviour of popular models, we measure their over-reliance on memorized information (the cause of hallucinations), and uncover important factors that exacerbate this behaviour. Lastly, we propose a simple method to mitigate over-reliance on parametric knowledge, which minimizes hallucination, and improves out-of-distribution generalization by 4% - 7%. Our findings demonstrate the importance for practitioners to evaluate model tendency to hallucinate rather than read, and show that our mitigation strategy encourages generalization to evolving information (i.e. time-dependent queries). To encourage these practices, we have released our framework for generating knowledge conflicts.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “…toxic/harmful behaviour (Sheng et al., 2019; Liang et al., 2021; Wallace et al., 2019), generate non-factual information (Maynez et al., 2020; Longpre et al., 2021; Devaraj et al., 2022), and other challenges in deployment and evaluation (Zellers et al., 2019; McGuffie & Newhouse, 2020;…”
Internet-augmented language models through few-shot prompting for open-domain question answering (2022, background): “, in retrieved evidence) knowledge poses challenges for retrievalbased models (Longpre et al., 2021).”
Survey of Hallucination in Natural Language Generation (2022, background): “[108] have discovered that such models prioritize parametric knowledge over the provided input.”
Towards Continual Knowledge Learning of Language Models (2021, background): “…suffer from hallucination, which means that they present false information as if it were correct, despite being given updated knowledge during inference, which worsens as the size of the LM increases (Longpre et al., 2021), making it more so important for implicit parameters to be renewed as well.”
Attributed Text Generation via Post-hoc Research and Revision (2022, background): “, 2021), or even contradicts the document (Longpre et al., 2021).”
RARR: Researching and Revising What Language Models Say, Using Language Models (2022, background): “, 2021), or even contradicts the document (Longpre et al., 2021).”
Making Retrieval-Augmented Language Models Robust to Irrelevant Context (2023, background): “A similar line of work focuses on when models should use parametric or retrieved knowledge, especially when there are conflicts (Longpre et al., 2021; Chen et al., 2022).”
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding (2023, methodology): “We use Exact Match (EM) as the evaluation metric for NQ-Swap, NQ and MemoTrap.”
RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models (2023, methodology): “Hallucination vs Models Table 3 illustrates in Table 4, there is a clear overall trend of an in-411 Table 6: The span-level detection performance for each baseline method across different tasks and different models. provide a response-level detection outcome.”
Knowledge Conflicts for LLMs: A Survey (2024, positive): “Longpre et al. (2021) reveal a tendency of models to over-rely on parametric knowledge.”
Tool Learning with Foundation Models (2023, neutral): “, 2020) may rely too heavily on their own knowledge after being fine-tuned on a specific dataset (Longpre et al., 2021).”
Prompting GPT-3 To Be Reliable (2022, positive): “Experiment Setup Our evaluation setup is inspired by Longpre et al. (2021), who reason about counterfactual scenarios.”
Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts (2023, methodology): “Notably, LLMs may respond with “Unknown” when no evidence is provided in the context, particularly in ChatGPT.”
When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (2022, background): “…chunks) enables much smaller models to match the performance of larger models (Izacard et al., 2022b; Khandelwal et al., 2020; Min et al., 2022), although Chen et al. (2022) and Longpre et al. (2021) show that even those models can ignore non-parametric knowledge and rely on parametric knowledge.”
RealTime QA: What's the Answer Right Now? (2022, background): “(Karpukhin et al., 2020); RAG: retrieval-augmented generation (Lewis et al., 2020b).”
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model (2022, methodology): “As a robustness check, we created an entity-perturbed version of the SQuAD dataset, similar to Longpre et al. (2021) and Yan et al. (2022), in which entity names were automatically substituted in the passages and gold answers.”
Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks (2021, background): “…the disjoint training 044 process in the prior work disregards the evidential- 045 ity of passages, leading to generation models that 046 ignore retrieved passages, leverage spurious cues, 047 and generate hallucinations when the context is not 048 evident (Longpre et al., 2021; Xu et al., 2021).”
FRUIT: Faithfully Reflecting Updated Information in Text (2021, background): “Whenever the provided evidence contradicts parametric knowledge, the model must prefer the evidence, which recent work has shown is difficult for pretrained language models (Krishna et al., 2021; Longpre et al., 2021).”