Conformer-Kernel with Query Term Independence for Document Retrieval
By Bhaskar Mitra, Sebastian Hofstätter, Hamed Zamani, Nick Craswell. 2020
22 citations recorded by the source corpus.
The Transformer-Kernel (TK) model has demonstrated strong reranking performance on the TREC Deep Learning benchmark---and can be considered to be an efficient (but slightly less effective) alternative to BERT-based ranking models. In this work, we extend the TK architecture to the full retrieval setting by incorporating the query term independence assumption. Furthermore, to reduce the memory complexity of the Transformer layers with respect to the input sequence length, we propose a new Conformer layer. We show that the Conformer's GPU memory requirement scales linearly with input sequence length, making it a more viable option when ranking long documents. Finally, we demonstrate that incorporating explicit term matching signal into the model can be particularly useful in the full retrieval setting. We present preliminary results from our work in this paper.
Subfields: Information Retrieval & Search, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Retrieval & Search
- Jimmy J. Lin
- Sebastian Hofstätter
- Allan Hanbury
- Rodrigo Nogueira
- Andrew Yates
- Nick Craswell
- Bhaskar Mitra
- Emine Yilmaz
In Natural Language Processing
- Jimmy J. Lin
- Rodrigo Nogueira
- Andrew Yates
- Sebastian Hofstätter
- Nick Craswell
- Bhaskar Mitra
- Allan Hanbury
- Emine Yilmaz
Citations in the corpus
- Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “student models [9], and other transformer architectures, including ground-up redesign efforts [14, 28] and adapting pretrained sequence-to-sequence models [8, 31].”
- TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime (2021, methodology): “To make it easier to work with this benchmark, we open-source a PyTorch [21] implementation3 of the Conformer-Kernel model [19, 20].”
- Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation (2020, methodology): “Several approaches aim to improve the efficiency of transformer models with windowed self-attention [16], using locality-sensitive hashing [22], replacing the self-attention with a local windowed and global attention [2] or by combining an efficient transformer-kernel model with a conformer layer [3”