Modularized Transfomer-based Ranking Framework

By Luyu Gao, Zhuyun Dai, Jamie Callan. 2020

49 citations recorded by the source corpus.

Recent innovations in Transformer-based ranking models have advanced the state-of-the-art in information retrieval. However, these Transformers are computationally expensive, and their opaque hidden states make it hard to understand the ranking process. In this work, we modularize the Transformer ranker into separate modules for text representation and interaction. We show how this design enables substantially faster ranking using offline pre-computed representations and light-weight online interactions. The modular design is also easier to interpret and sheds light on the ranking process in Transformer rankers.

Subfields: Artificial Intelligence, Information Systems, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Signal Processing

Citations in the corpus

RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering (2020, neutral): “The re-rankers (Khattab and Zaharia, 2020; Gao et al., 2020) with light weight interaction based on the representations of dense retrievers have been studied.”
Condenser: a Pre-training Architecture for Dense Retrieval (2021, background): “While there are works on efficient cross-encoder (Gao et al., 2020; MacAvaney et al., 2020), such models are still too costly for full”
COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List (2021, background): “Lighter deep LM rankers are developed (MacAvaney et al., 2020; Gao et al., 2020), but their cross attention operations are still too expensive for fullcollection retrieval.”
Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval (2021, background): “The top100 are reranked by a fast modular reranker (Gao et al., 2020).”
RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking (2021, background): “Apart from separately considering the above two tasks, it has been proved that passage retrieval and passage re-ranking are actually highly related and dependent (Huang et al., 2020; Gao et al., 2020; Khattab and Zaharia, 2020).”
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (2021, background): “Several other systems leverage multi-vector representations, including Poly-encoders (Humeau et al., 2020), PreTTR (MacAvaney et al., 2020), and MORES (Gao et al., 2020), but they target attention-based re-ranking as opposed to ColBERT’s scalable MaxSim-based interaction.”