Long Document Ranking with Query-Directed Sparse Transformer
By Jyun-Yu Jiang, Chenyan Xiong, Chia-Jung Lee. 2020
31 citations recorded by the source corpus.
The computing cost of transformer self-attention often necessitates breaking long documents to fit in pretrained models in document ranking tasks. In this paper, we design Query-Directed Sparse attention that induces IR-axiomatic structures in transformer self-attention. Our model, QDS-Transformer, enforces the principle properties desired in ranking: local contextualization, hierarchical representation, and query-oriented proximity matching, while it also enjoys efficiency from sparsity. Experiments on four fully supervised and few-shot TREC document ranking benchmarks demonstrate the consistent and robust advantage of QDS-Transformer over previous approaches, as they either retrofit long documents into BERT or use sparse attention without emphasizing IR principles. We further quantify the computing complexity and demonstrates that our sparse attention with TVM implementation is twice more efficient that the fully-connected self-attention. All source codes, trained model, and predictions of this work are available at https://github.com/hallogameboy/QDS-Transformer.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Nick Craswell
- Bhaskar Mitra
- Jimmy J. Lin
- Emine Yilmaz
- Daniel Fernando Campos
- Sebastian Hofstätter
- Allan Hanbury
- Hamed Zamani
Citations in the corpus
- MS MARCO: Benchmarking Ranking Models in the Large-Data Regime (2021, methodology): “Similar strategies of finetuning on MS MARCO and evaluating on Robust04, GOV2 [10], and ClueWeb [11] have been employed in other recent studies [30, 37, 40, 91, 92], sometimes in combination with weak”