Pretrained Transformers for Text Ranking: BERT and Beyond

By Jimmy J. Lin, Rodrigo Nogueira, Andrew Yates. 2020

768 citations recorded by the source corpus.

The goal of text ranking is to generate an ordered list of texts retrieved from a corpus in response to a query for a particular task. Although the most common formulation of text ranking is search, instances of the task can also be found in many text processing applications. This tutorial provides an overview of text ranking with neural network architectures known as transformers, of which BERT (Bidirectional Encoder Representations from Transformers) is the best-known example. These models produce high quality results across many domains, tasks, and settings. This tutorial, which is based on the preprint of a forthcoming book to be published by Morgan and & Claypool under the Synthesis Lectures on Human Language Technologies series, provides an overview of existing work as a single point of entry for practitioners who wish to deploy transformers for text ranking in real-world applications and researchers who wish to pursue work in this area. We cover a wide range of techniques, grouped into two categories: transformer models that perform reranking in multi-stage ranking architectures and learned dense representations that perform ranking directly.

Subfields: Artificial Intelligence, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Few-shot Learning with Retrieval Augmented Language Models (2022, neutral): “We refer the reader to Yates et al. (2021) for a survey of dense retrieval techniques.”
Unsupervised Dense Information Retrieval with Contrastive Learning (2021, background): “This is not an exhaustive review, and we refer the reader to Manning et al. (2008), Mitra et al. (2018) and Lin et al. (2020) for a more complete introduction to the ﬁeld.”
Learning Dense Representations of Phrases at Scale (2020, methodology): “This type of dense retrieval has been also studied for sentence and passage retrieval (Humeau et al., 2019; Karpukhin et al., 2020) (see Lin et al., 2020 for recent advances in dense retrieval).”
Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations (2021, background): “The advent of pretrained transformers has led to many exciting recent developments in information retrieval [15].”
Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations (2021, background): “The advent of pretrained transformers has led to many exciting recent developments in information retrieval [16].”
A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques (2021, neutral): “See Lin et al. (2020) for a survey.”
Phrase Retrieval Learns Passage Retrieval, Too (2021, methodology): “We refer to Lin et al. (2020) for a comprehensive overview of neural text retrieval methods.”
BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models (2021, background): “Recently, deep learning and in particular pre-trained Transformer models like BERT [12] have become popular in information retrieval [75].”
TILDE: Term Independent Likelihood moDEl for Passage Re-ranking (2021, positive): “In this paper, we focus on deep query likelihood models [8, 33] that typically rely on a transformer-based decoder as the model architecture [14].”
MS MARCO: Benchmarking Ranking Models in the Large-Data Regime (2021, background): “As pointed out by some researchers [44], to a large extent, the rapid progress made in the IR community would not have been possible without MS MARCO.”
Towards Unsupervised Dense Information Retrieval with Contrastive Learning (2021, background): “This is not an exhaustive review, and we refer the reader to Manning et al. (2008), Mitra et al. (2018) and Lin et al. (2020) for a more complete introduction to the ﬁeld.”
TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime (2021, background): “The overall motivation for the TREC Deep Learning Track is to allow benchmarking of machine learning based ranking approaches [15, 16, 18] against traditional IR methods, in a setting where large training data is available.”
Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval (2021, background): “We have not yet systematically explored the broad design space of what Lin et al. (2020) calls “multi-step ﬁne-tuning strategies”, par-alleling the explorations of Shi et al. (2020) in the context of transformer-based reranking models.”
M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation (2024, positive): “One major application of embedding models is neural retrieval (Lin et al., 2022).”
Task-aware Retrieval with Instructions (2022, background): “ratives only marginally outperforms the baselines with titles only (Walker et al., 1998; Yates et al., 2021).”
Boosted Dense Retriever (2021, neutral): “See Yates et al. (2021) for a survey.”
Distilling Dense Representations for Ranking using Tightly-Coupled Teachers (2020, methodology): “The ad hoc retrieval task can be viewed as a text ranking problem; here, we adopt the formulation of Lin et al. (2020).”
One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval (2021, background): “Answering multilingual questions requires retrieving evidence from knowledge sources of other languages than the original question since many languages have limited reference documents or the question sometimes inquires about concepts from other cultures (Asai et al., 2021; Lin et al., 2020).”