ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT

By Omar Khattab, Matei A. Zaharia. 2020

2,316 citations recorded by the source corpus.

Recent progress in Natural Language Understanding (NLU) is driving fast-paced advances in Information Retrieval (IR), largely owed to fine-tuning deep language models (LMs) for document ranking. While remarkably effective, the ranking models based on these LMs increase computational cost by orders of magnitude over prior approaches, particularly as they must feed each query-document pair through a massive neural network to compute a single relevance score. To tackle this, we present ColBERT, a novel ranking model that adapts deep LMs (in particular, BERT) for efficient retrieval. ColBERT introduces a late interaction architecture that independently encodes the query and the document using BERT and then employs a cheap yet powerful interaction step that models their fine-grained similarity. By delaying and yet retaining this fine-granular interaction, ColBERT can leverage the expressiveness of deep LMs while simultaneously gaining the ability to pre-compute document representations offline, considerably speeding up query processing. Crucially, ColBERT's pruning-friendly interaction mechanism enables leveraging vector-similarity indexes for end-to-end retrieval directly from millions of documents. We extensively evaluate ColBERT using two recent passage search datasets. Results show that ColBERT's effectiveness is competitive with existing BERT-based models (and outperforms every non-BERT baseline), while executing two orders-of-magnitude faster and requiring up to four orders-of-magnitude fewer FLOPs per query.

Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jimmy J. Lin
  2. Luke S. Zettlemoyer
  3. Mohit Bansal
  4. Jianfeng Gao
  5. Hannaneh Hajishirzi
  6. Graham Neubig
  7. Percy Liang
  8. Sebastian Riedel

In Information Systems

  1. Jimmy J. Lin
  2. Jamie Callan
  3. Rodrigo Nogueira
  4. Sebastian Hofstätter
  5. Allan Hanbury
  6. Zhuyun Dai
  7. Nick Craswell
  8. Bhaskar Mitra

In Information Retrieval & Search

  1. Jimmy J. Lin
  2. Jamie Callan
  3. Sebastian Hofstätter
  4. Luyu Gao
  5. Omar Khattab
  6. Allan Hanbury
  7. Wen-tau Yih
  8. Sebastian Riedel

Citations in the corpus