Pre-training Tasks for Embedding-based Large-scale Retrieval

By Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, Sanjiv Kumar. 2020

327 citations recorded by the source corpus.

We consider the large-scale query-document retrieval problem: given a query (e.g., a question), return the set of relevant documents (e.g., paragraphs containing the answer) from a large document corpus. This problem is often solved in two steps. The retrieval phase first reduces the solution space, returning a subset of candidate documents. The scoring phase then re-ranks the documents. Critically, the retrieval algorithm not only desires high recall but also requires to be highly efficient, returning candidates in time sublinear to the number of documents. Unlike the scoring phase witnessing significant advances recently due to the BERT-style pre-training tasks on cross-attention models, the retrieval phase remains less well studied. Most previous works rely on classic Information Retrieval (IR) methods such as BM-25 (token matching + TF-IDF weights). These models only accept sparse handcrafted features and can not be optimized for different downstream tasks of interest. In this paper, we conduct a comprehensive study on the embedding-based retrieval models. We show that the key ingredient of learning a strong embedding-based Transformer model is the set of pre-training tasks. With adequately designed paragraph-level pre-training tasks, the Transformer models can remarkably improve over the widely-used BM-25 as well as embedding models without Transformers. The paragraph-level pre-training tasks we studied are Inverse Cloze Task (ICT), Body First Selection (BFS), Wiki Link Prediction (WLP), and the combination of all three.

Subfields: Artificial Intelligence, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

Citations in the corpus

Distilling Knowledge from Reader to Retriever for Question Answering (2020, methodology): “This pre-training scheme was later evaluated for ad-hoc document retrieval by Chang et al. (2020).”
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, methodology): “Dense Retrieval (DR) using learned distributed representations is a promising direction to overcome this sparse retrieval bottleneck [9, 10, 11, 12, 13, 14, 15]: The representation space is fully learnable and can leverage the strength of pretraining, while the retrieval operation is sufﬁciently…”
Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering (2020, methodology): “In order to build effective paragraph encoders tailed for the paragraph retrieval in open-domain QA, more recent studies (Lee et al., 2019; Chang et al., 2020; Guu et al., 2020) propose to pretrain Trans-former encoders (Vaswani et al., 2017) using objectives that simulate the semantic matching…”
Unsupervised Dense Information Retrieval with Contrastive Learning (2021, neutral): “While showing promising results as pre-training (Chang et al., 2020; Sachan et al., 2021a), this approach still lags behind BM25 when used as a zero-shot retriever.”
Domain-matched Pre-training Tasks for Dense Retrieval (2021, background): “Chang et al. (2020) investigate several artiﬁcial tasks for training dense retrieval models, including ICT and BFS, showing improvements over no pre-training.”
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering (2020, methodology): “Several recent studies (Karpukhin et al., 2020; Luan et al., 2020; Chang et al., 2020; Henderson et al., 2017) tried to address the ﬁrst challenge by designing complicated sampling mechanism to generate hard negatives.”
Complementing Lexical Retrieval with Semantic Residual Embedding (2020, background): “Note that although BERT-based embeding models have been tested on several sentence-level tasks (Reimers and Gurevych, 2019; Chang et al., 2020), its effectiveness for passage retrieval remains to be studied.”
LinkBERT: Pretraining Language Models with Document Links (2022, background): “Chang et al. (2020); Asai et al. (2020); Seonwoo et al. (2021) use Wikipedia hyperlinks to train retrievers for open-domain question answering.”
Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation (2020, background): “g. Average ColBERT BERT DOT BERT CAT TK Figure 1: Raw query-passage pair scores during training of different ranking models. The margin between the positive and negative samples is shaded. Tower-BERT [4], BERT-Siamese [42], or TwinBERT [25].1 Each approach has unique characteristics that make them s”
MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale (2020, methodology): “For instance, combining our approach with additional pre-training objectives such as the Inverse Cloze Task (Chang et al., 2020) could substantially increase the amount of training data for the large quantity of smaller forums.”
Towards General Text Embeddings with Multi-stage Contrastive Learning (2023, methodology): “Building upon the ideas presented in (Chang et al., 2020), some researchers have also put forth methods for constructing higher-quality positive pairs using the web link topology for retriever pre-training (Zhou et al., 2022), a technique that proves effective in zero-shot scenarios.”
In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval (2021, methodology): “The bi-encoder design has been widely adopted for dense retrieval (Lee et al., 2019; Chang et al., 2020; Guu et al., 2020; Karpukhin et al., 2020; Luan et al., 2021; Qu et al., 2020; Xiong et al., 2021), where queries and passages are encoded in a low-dimensional space.”
PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2020, methodology): “al [6], ICT outperforms BFS and WLP task.”
Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval (2021, methodology): “Building upon the advancements of pre-trained language models (LM; Devlin et al. (2019); Liu et al. (2019)), dense retrieval has become an effective paradigm for text retrieval (Lee et al., 2019; Chang et al., 2020; Karpukhin et al., 2020; Qu et al., 2021).”
Dialog Inpainting: Turning Documents into Dialogs (2022, methodology): “Data augmentation has been widely used to improve the performance of document retrieval and related tasks (Lee et al., 2019; Chang et al., 2020; Gao et al., 2021) by using training data more efficiently.”
NewsEmbed: Modeling News through Pre-trained Document Representations (2021, background): “• Text embedding: Token embeddings in the title and keywords are averaged to represent a document [10].”