Document Ranking with a Pretrained Sequence-to-Sequence Model

By Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, Jimmy J. Lin. 2020

821 citations recorded by the source corpus.

This work proposes the use of a pretrained sequence-to-sequence model for document ranking. Our approach is fundamentally different from a commonly adopted classification-based formulation based on encoder-only pretrained transformer architectures such as BERT. We show how a sequence-to-sequence model can be trained to generate relevance labels as “target tokens”, and how the underlying logits of these target tokens can be interpreted as relevance probabilities for ranking. Experimental results on the MS MARCO passage ranking task show that our ranking approach is superior to strong encoder-only models. On three other document retrieval test collections, we demonstrate a zero-shot transfer-based approach that outperforms previous state-of-the-art models requiring in-domain cross-validation. Furthermore, we find that our approach significantly outperforms an encoder-only architecture in a data-poor setting. We investigate this observation in more detail by varying target tokens to probe the model’s use of latent knowledge. Surprisingly, we find that the choice of target tokens impacts effectiveness, even for words that are closely related semantically. This finding sheds some light on why our sequence-to-sequence formulation for document ranking is effective. Code and models are available at pygaggle.ai.

Subfields: Artificial Intelligence, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

Citations in the corpus

TTTTTackling WinoGrande Schemas (2020, neutral): “This general problem setup allows us to choose the target tokens, which may have an impact on the prediction accuracy [4].”
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, methodology): “…for the “Text-to-Text Transfer Transformer” (T5) model released by Raﬀel et al. (2019) have been used to achieve state-of-the-art results on many benchmarks (Khashabi et al., 2020; Roberts et al., 2020; Kale, 2020; Izacard and Grave, 2020; Nogueira et al., 2020; Narang et al., 2020, etc.).”
Re2G: Retrieve, Rerank, Generate (2022, methodology): “Transformers may be applied as classiﬁers to each query and passage pair independently [Nogueira and Cho, 2019] or as generators to produce labels for passages in a sequence-to-sequence model [Nogueira et al., 2020].”
R2-D2: A Modular Baseline for Open-Domain Question Answering (2021, methodology): “Another document ranking approach uses the seq2seq model to generate a true or false answer to the document’s relevance to the query (Nogueira et al., 2020).”
Autoregressive Entity Retrieval (2020, methodology): “Nogueira et al. (2020) propose to use a sequence-to-sequence model to re-rank document.”
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering (2020, neutral): “0 4 expando-mono-duo-T5 (Nogueira et al., 2020) May 19, 2020 40.”
Designing Templates for Eliciting Commonsense Knowledge from Pretrained Sequence-to-Sequence Models (2020, methodology): “To deal with these cases, we apply a softmax over the logits of the pair of predeﬁned target tokens, similar to Nogueira et al. (2020).”
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting (2023, background): “Existing results usually signiﬁcantly underperform well-trained baseline rankers (e.g., Nogueira et al. (2020); Zhuang et al. (2023)).”
BERT-QE: Contextualized Query Expansion for Document Re-ranking (2020, methodology): “Inspired by the recent advances of pre-trained contextualized models like BERT for the ranking task (Yilmaz et al., 2019; Nogueira et al., 2020), this work attempts to develop query expansion models based on BERT with the goal of more effectively using the relevant information from PRF.”
Video Understanding as Machine Translation (2020, background): “[35] and rank the possible choices in textual form (i.”
A Little Bit Is Worse Than None: Ranking with Limited Training Data (2020, methodology): “Nogueira et al. (2020) conﬁrmed this ﬁnding and further improved zero-shot effectiveness on Robust04 by ﬁne-tuning T5 (Raffel et al., 2020) on the MS MARCO passage dataset (Bajaj et al., 2018).”