Retrieval-based Language Models and Applications
By Akari Asai, Sewon Min, Zexuan Zhong, Danqi Chen. 2023
142 citations recorded by the source corpus.
Retrieval-based language models (LMs) have shown impressive performance on diverse NLP tasks. In this tutorial, we will provide a comprehensive and coherent overview of recent advances in retrieval-based LMs. We will start by providing preliminaries covering the foundation of LMs (e.g., masked LMs, autoregressive LMs) and retrieval systems (e.g., nearest-neighbor search). We will then detail recent progress in retrieval-based models, focusing on their model architectures and learning approaches. Finally, we will show how retrieval-based LMs are adapted to downstream applications, and extended to multilingual and multi-modal settings. Finally, we will use an exercise to showcase the effectiveness of retrieval-based LMs.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Retrieval-Augmented Generation, Large Language Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Hannaneh Hajishirzi
- Akari Asai
- Sean Welleck
- Yizhong Wang
- Luke S. Zettlemoyer
- Hailey Schoelkopf
- Hong-Liu Cao
- Mohit Bansal
In Computer Vision and Pattern Recognition
- Hannaneh Hajishirzi
- Akari Asai
- Yizhong Wang
- Sean Welleck
- Hong-Liu Cao
- Stella Biderman
- Albert Qiaochu Jiang
- Zhangir Azerbayev
In Retrieval-Augmented Generation
In Large Language Models
- Hailey Schoelkopf
- Sean Welleck
- Stella Biderman
- Yizhong Wang
- Albert Qiaochu Jiang
- Zhangir Azerbayev
- Hong-Liu Cao
- Marco Dos Santos
Citations in the corpus
- Llemma: An Open Language Model For Mathematics (2023, background): “, 2023), retrieval augmentation (Min et al., 2023; Asai et al., 2023), and other techniques.”
- Retrieval-Augmented Generation for Large Language Models: A Survey (2023, methodology): “Self-RAG [Asai et al. , 2023] introduces “reflection to-kens” that allow the model to introspect its outputs.”
- A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models (2024, unknown): “For RAG, the choice of retrieval granularity can significantly impact the overall performance of the model in terms of effectiveness and efficiency as they determine the saving space for the database as well as the computational cost for searching [4].”
- A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, unknown): “In the general meaning learning domain, a transparent model is typically characterized by decision-making processes akin to those of white-box models, e.g., decision tree-based models or linear regression models.”
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (2023, methodology): “Retrieval-Augmented Generation (RAG) methods (Figure 1 left; Lewis et al. 2020; Guu et al. 2020) augment the input of LLMs with relevant retrieved passages, reducing factual errors in knowledge-intensive tasks (Ram et al., 2023; Asai et al., 2023a).”
- Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark (2024, background): “This is mainly due to the fact that these LLM based applications are heavily dependent on the high quality text embeddings for tasks like vector search, a process where the most relevant documents are retrieved for LLM Question Answering (QA) [12], [14].”