Teaching language models to support answers with verified quotes

By Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving and 1 more. 2022

349 citations recorded by the source corpus.

Recent large language models often answer factual questions correctly. But users can't trust any given claim a model makes without fact-checking, because language models can hallucinate convincing nonsense. In this work we use reinforcement learning from human preferences (RLHP) to train"open-book"QA models that generate answers whilst also citing specific evidence for their claims, which aids in the appraisal of correctness. Supporting evidence is drawn from multiple documents found via a search engine, or from a single user-provided document. Our 280 billion parameter model, GopherCite, is able to produce answers with high quality supporting evidence and abstain from answering when unsure. We measure the performance of GopherCite by conducting human evaluation of answers to questions in a subset of the NaturalQuestions and ELI5 datasets. The model's response is found to be high-quality 80\% of the time on this Natural Questions subset, and 67\% of the time on the ELI5 subset. Abstaining from the third of questions for which it is most unsure improves performance to 90\% and 80\% respectively, approaching human baselines. However, analysis on the adversarial TruthfulQA dataset shows why citation is only one part of an overall strategy for safety and trustworthiness: not all claims supported by evidence are true.

Subfields: Artificial Intelligence, Health Informatics, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

OpenAssistant Conversations - Democratizing Large Language Model Alignment (2023, background): “…ﬁne-tuning of language models that has recently emerged as a promising approach to making large language models more in line with human preferences by generating more desirable outputs based on explicitly collected human preference data [6, 7, 8, 9, 10, 11] and thus making them more useful.”
Pretraining Language Models with Human Preferences (2023, background): “, 2022) or reinforcement learning from human feedback (RLHF; Ziegler et al., 2019; Ouyang et al., 2022; Bai et al., 2022; Menick et al., 2022), but this strategy is also limited by the fact that large LMs are quite resistant to forgetting their training data (an effect that increases with model size”
Internet-augmented language models through few-shot prompting for open-domain question answering (2022, background): “, 2022) or focus on optimizing performance using more intensive techniques like fine-tuning and reinforcement learning (Nakano et al., 2021; Menick et al., 2022).”
Scaling Laws for Reward Model Overoptimization (2022, background): “…to train language models to perform a variety of complex tasks that are hard to judge automatically, including summarization [Stiennon et al., 2020], question-answering [Nakano et al., 2021, Menick et al., 2022], and general assistance [Ouyang et al., 2022, Bai et al., 2022, Glaese et al., 2022].”
Improving alignment of dialogue agents via targeted human judgements (2022, methodology): “Multi-turn supported and plausible evaluation We assess Sparrow’s responses and accompanying evidence through human evaluation, using the metrics of supported and plausible as defined in section 2.4 and GopherCite (Menick et al., 2022).”
Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “In the rest of the paper, we will focus on target distributions modeling three types of preferences prominently employed in recent literature about GDC (Khalifa et al., 2021) and RLHF (Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022; Menick et al., 2022; Bai et al., 2022a).”
Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning (2022, background): “In the future this assumption can be relaxed, for example by extracting the necessary information through search (Lazaridou et al., 2022; Menick et al., 2022).”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “[72] Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, and Nat McAleese.”
Taxonomy of Risks posed by Language Models (2022, background): “Mitigation strategies in LMs include increasing model size and responsible release strategies, forcing LMs to provide in-line references for statements [128], architectural innovations such as retrieval models [25, 134] and adaptive models that learn dynamically over time [112], and”
Large Language Models for Information Retrieval: A Survey (2023, neutral): “It is crucial to address these challenges to effectively realize the potential of LLMs in this paradigm.”
A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models (2024, positive): “Furthermore, white-box generators allow parameter optimization, which can be trained to adapt to different retrieval and augmentation approaches for a better performance of generation.”
Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons (2023, neutral): “…(Brown et al., 2019; Shin et al., 2023) and language models (Ziegler et al., 2019; Stiennon et al., 2020; Wu et al., 2021; Nakano et al., 2021; Ouyang et al., 2022; Menick et al., 2022; Glaese et al., 2022; Gao et al., 2022; Bai et al., 2022a; Ganguli et al., 2022; Ramamurthy et al., 2022).”
Understanding the Effects of RLHF on LLM Generalisation and Diversity (2023, methodology): “More RLHF and SFT details Often, SFT and RLHF are combined by performing SFT followed by RLHF (Glaese et al., 2022; Menick et al., 2022; Nakano et al., 2022; Ouyang et al., 2022).”
A Comprehensive Overview of Large Language Models (2023, neutral): “Similar to the RLHF training style, a reward model is trained to rank generated responses containing web citations in answers to questions, which is later used to train the model, as in GopherCite [146], WebGPT [147], and Sparrow [148].”