Delaying Interaction Layers in Transformer-based Encoders for Efficient Open Domain Question Answering
By Wissam Siblini, Mohamed Challal, Charlotte Pasqual. 2020
3 citations recorded by the source corpus.
Open Domain Question Answering (ODQA) on a large-scale corpus of documents (e.g. Wikipedia) is a key challenge in computer science. Although transformer-based language models such as Bert have shown on SQuAD the ability to surpass humans for extracting answers in small passages of text, they suffer from their high complexity when faced to a much larger search space. The most common way to tackle this problem is to add a preliminary Information Retrieval step to heavily filter the corpus and only keep the relevant passages. In this paper, we propose a more direct and complementary solution which consists in applying a generic change in the architecture of transformer-based models to delay the attention between subparts of the input and allow a more efficient management of computations. The resulting variants are competitive with the original models on the extractive task and allow, on the ODQA setting, a significant speedup and even a performance improvement in many cases.
Subfields: Natural Language Processing, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Natural Language Processing
- Jinhyuk Lee
- Danqi Chen
- Jaewoo Kang
- Mujeen Sung
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Mohit Bansal
- Percy Liang
In Efficiency & Compression
Citations in the corpus
- Learning Dense Representations of Phrases at Scale (2020, background): “, 2020) and DilBERT (Siblini et al., 2020), where they only allow a late interaction (cross-attention) in the last few layers of BERT.”