ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
By Omar Khattab, Matei A. Zaharia. 2020
2,316 citations recorded by the source corpus.
Recent progress in Natural Language Understanding (NLU) is driving fast-paced advances in Information Retrieval (IR), largely owed to fine-tuning deep language models (LMs) for document ranking. While remarkably effective, the ranking models based on these LMs increase computational cost by orders of magnitude over prior approaches, particularly as they must feed each query-document pair through a massive neural network to compute a single relevance score. To tackle this, we present ColBERT, a novel ranking model that adapts deep LMs (in particular, BERT) for efficient retrieval. ColBERT introduces a late interaction architecture that independently encodes the query and the document using BERT and then employs a cheap yet powerful interaction step that models their fine-grained similarity. By delaying and yet retaining this fine-granular interaction, ColBERT can leverage the expressiveness of deep LMs while simultaneously gaining the ability to pre-compute document representations offline, considerably speeding up query processing. Crucially, ColBERT's pruning-friendly interaction mechanism enables leveraging vector-similarity indexes for end-to-end retrieval directly from millions of documents. We extensively evaluate ColBERT using two recent passage search datasets. Results show that ColBERT's effectiveness is competitive with existing BERT-based models (and outperforms every non-BERT baseline), while executing two orders-of-magnitude faster and requiring up to four orders-of-magnitude fewer FLOPs per query.
Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jimmy J. Lin
- Luke S. Zettlemoyer
- Mohit Bansal
- Jianfeng Gao
- Hannaneh Hajishirzi
- Graham Neubig
- Percy Liang
- Sebastian Riedel
In Information Systems
- Jimmy J. Lin
- Jamie Callan
- Rodrigo Nogueira
- Sebastian Hofstätter
- Allan Hanbury
- Zhuyun Dai
- Nick Craswell
- Bhaskar Mitra
In Information Retrieval & Search
- Jimmy J. Lin
- Jamie Callan
- Sebastian Hofstätter
- Luyu Gao
- Omar Khattab
- Allan Hanbury
- Wen-tau Yih
- Sebastian Riedel
Citations in the corpus
- Dense Passage Retrieval for Open-Domain Question Answering (2020, background): “(1)) becomes a good ranking function for retrieval is essentially a metric learning problem (Kulis, 2013).”
- End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering (2021, background): “There are two broad classes of neural retrievers based on the number of embeddings computed for a document: dual encoders (Yih et al., 2011, Lee et al., 2019) and multivector encoders (Khattab and Zaharia, 2020, Luan et al., 2021).”
- Learning Dense Representations of Phrases at Scale (2020, background): “While a similar problem is observed in learning dual encoders for passage representations (Humeau et al., 2019; Khattab and Zaharia, 2020), phrase representations are even more difficult to learn due to the fine-grained representations.”
- Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2024, methodology): “This includes exploring dense retrievers trained directly on the multihop retrieval task, such as those based on ColBERT (Khattab and Zaharia, 2020), or SimCSE (Gao et al., 2021) architectures.”
- Relevance-guided Supervision for OpenQA with ColBERT (2020, methodology): “For this, we train ColBERT as in Khattab and Zaharia (2020) on the MS MARCO Ranking dataset, a natural choice for transfer learning to OpenQA.”
- Learning To Retrieve Prompts for In-Context Learning (2021, background): “Retrieval Research on training dense retrievers has skyrocketed recently, propelled by interest in open-domain question answering (Chen et al., 2017; Lee et al., 2019; Karpukhin et al., 2020; Guu et al., 2020; Khattab and Zaharia, 2020; Qu et al., 2021).”
- Multi-Task Retrieval for Knowledge-Intensive Tasks (2021, background): “Of these, ColBERT (Khattab and Zaharia, 2020) and ME-BERT (Luan et al., 2020) are two representative examples.”
- Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations (2021, methodology): “Row (3a) shows the results of combining TCT-ColBERTwith BM25 bag-of-words search over the original texts, while row (3b) shows results that combine document expansion using doc2query with the T5 model.”
- Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations (2021, methodology): “Row (3a) shows the results of combining TCT-ColBERT with BM25 bag-of-words search over the original texts, while row (3b) shows results that combine document expansion using doc2query with the T5 model.”
- RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering (2020, neutral): “The re-rankers (Khattab and Zaharia, 2020; Gao et al., 2020) with light weight interaction based on the representations of dense retrievers have been studied.”
- Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation (2022, methodology): “Most modern information retrieval systems are based on the index-then-retrieve pipeline where documents are first encoded and stored in an inverted index [10, 13, 15, 26, 34, 55, 58] or a nearest neighbor search index [7, 14, 20, 21, 25, 29, 49, 51] and search results are then constructed based on a”