Internet-augmented language models through few-shot prompting for open-domain question answering
By Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, N. Grigorev. 2022
177 citations recorded by the source corpus.
In this work, we aim to capitalize on the unique few-shot capabilities of large-scale language models (LSLMs) to overcome some of their challenges with respect to grounding to factual and up-to-date information. Motivated by semi-parametric language models (LMs), which ground their decisions in external retrieved evidence, we use few-shot prompting to learn to condition LMs on information returned from the web using Google Search, a broad and constantly updated knowledge source. Our approach does not involve fine-tuning or learning additional parameters, thus making it applicable to any LM, offering therefore a strong baseline. Indeed, we find that LMs conditioned on the web surpass performance of closed-book models of similar, or even larger, model sizes in open-domain question answering. Finally, we find that increasing the inference-time compute of models, achieved via using multiple retrieved evidences to generate multiple answers followed by a reranking stage that uses scores generated by the same LMs, leads to better performance and alleviates lower performance of smaller few-shot LMs. All in all, our findings suggest that it might be beneficial to slow down the race towards the biggest model and instead shift attention towards finding more effective ways to use models, including but not limited to, better prompting or increasing inference-time compute.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Large Language Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Yejin Choi
- Hannaneh Hajishirzi
- Quoc V. Le
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Mohit Bansal
- Trevor Darrell
- Dacheng Tao
- Jianfeng Gao
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
In Retrieval-Augmented Generation
- Sebastian Riedel
- Matei A. Zaharia
- Christopher Potts
- Omar Khattab
- Wen-tau Yih
- Fabio Petroni
- Patrick Lewis
- Chenguang Zhu
In Large Language Models
- Timo Schick
- Hinrich Schütze
- Sebastian Riedel
- Fabio Petroni
- Denny Zhou
- Quoc V. Le
- Ashish Sabharwal
- Wen-tau Yih
Citations in the corpus
- Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “Lazaridou et al. (2022) explore NaturalQuestions in a 15-shot setup using Gopher, augmenting questions with 50 passages retrieved using Google Search.”
- Augmented Language Models: a Survey (2023, background): “…with a few human-written in-context demonstrations of a desired behaviour is a common approach both for teaching them to reason (Wei et al., 2022c;b; Suzgun et al., 2022; Press et al., 2022) and for teaching them to use tools and act (Gao et al., 2022; Lazaridou et al., 2022; Yao et al., 2022b).”
- Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, background): “Term Weighting Virtually all modern approaches to string-matching-based sparse retrieval make use of a bag-of-words assumption, indexing documents with an inverted index , a data structure mapping terms to documents or, more generally, locations in a corpus (Robertson and Zaragoza, 2009).”
- Improving alignment of dialogue agents via targeted human judgements (2022, methodology): “We bootstrap from an initial evidence-supported dialogue model by prompting (Lazaridou et al., 2022; Menick et al., 2022).”
- WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, background): “Several papers have explored the use of the web for information extraction [34] and retrieval [1], question answering [57, 25], dialog [45], and training language models on webtext [2].”
- Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning (2022, background): “In the future this assumption can be relaxed, for example by extracting the necessary information through search (Lazaridou et al., 2022; Menick et al., 2022).”
- Teaching language models to support answers with verified quotes (2022, methodology): “Three closely related pieces of work have recently been released (Lazaridou et al., 2022; Nakano et al., 2021; Thoppilan et al., 2022).”
- Tool Learning with Foundation Models (2023, unknown): “Later works have gone beyond local repositories by leveraging the entire web as the knowledge source, which allows for improved temporal generalization and higher factual accuracy (Piktus et al., 2021; Lazaridou et al., 2022; Menick et al., 2022).”
- DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023, background): “In-context In-context learning methods now routinely invoke tools, leading to LM pipelines that use retrieval (Chen et al., 2017; Lewis et al., 2020; Guu et al., 2020; Lazaridou et al., 2022; Izacard et al., 2022), and APIs (Nakano et al., 2021).”
- A Survey of Large Language Models (2023, unknown): “By incorporating the extracted relevant information into the context [360, 361], LLMs can acquire new factual knowledge and perform better on relevant tasks.”
- Generate rather than Retrieve: Large Language Models are Strong Context Generators (2022, background): “This is actually the same setting as “zero-shot” reading comprehension, as widely studied in existing works (Brown et al., 2020; Lazaridou et al., 2022).”
- Large Language Models for Information Retrieval: A Survey (2023, unknown): “For example, REALM [166] adopts this strategy by directly attending the document contents to the original queries to predict the final answers based on masked language modeling.”
- RAFT: Adapting Language Model to Domain Specific RAG (2024, background): “Indeed, adapting LLMs to the specialized domains (e.g., recent news, enterprise private documents, or program resources constructed after the training cutoff) is essential to many emerging applications (Vu et al., 2023; Lazaridou et al., 2022) and is the focus of this work.”
- A Comprehensive Overview of Large Language Models (2023, unknown): “We identify critical information in Table I and II such as architecture, training strategies, and pipelines that improve LLMs’ performance or other abilities acquired because of changes mentioned in section III.”
- A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models (2024, unknown): “Furthermore, white-box generators allow parameter optimization, which can be trained to adapt to different retrieval and augmentation approaches for a better performance of generation.”
- FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation (2023, methodology): “…augments LLM S ’ prompts with knowledge retrieved from search engines for real-time alignment to current and factual information (Nakano et al., 2021; Lazaridou et al., 2022; Menick et al., 2022; Yao et al., 2022; Press et al., 2022; Khattab et al., 2022; Schick et al., 2023; Luo et al., 2023).”