DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering

By Qingqing Cao, Harsh Trivedi, Aruna Devi Balasubramanian, Niranjan Balasubramanian. 2020

71 citations recorded by the source corpus.

Transformer-based QA models use input-wide self-attention – i.e. across both the question and the input passage – at all layers, causing them to be slow and memory-intensive. It turns out that we can get by without input-wide self-attention at all layers, especially in the lower layers. We introduce DeFormer, a decomposed transformer, which substitutes the full self-attention with question-wide and passage-wide self-attentions in the lower layers. This allows for question-independent processing of the input text representations, which in turn enables pre-computing passage representations reducing runtime compute drastically. Furthermore, because DeFormer is largely similar to the original model, we can initialize DeFormer with the pre-training weights of a standard transformer, and directly fine-tune on the target QA dataset. We show DeFormer versions of BERT and XLNet can be used to speed up QA by over 4.3x and with simple distillation-based losses they incur only a 1% drop in accuracy. We open source the code at https://github.com/StonyBrookNLP/deformer.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

Citations in the corpus

Learning Dense Representations of Phrases at Scale (2020, background): “, 2020) as well as Deformer (Cao et al., 2020) and DilBERT (Siblini et al.”
Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “…layers in language models focus on local phenomena while higher layers focus on global ones (using probing classiﬁers and other methods) motivated Cao et al. (2020) to decouple a question-answering model, such that lower layers process the question and the passage independently and higher…”
Probing Classifiers: Promises, Shortcomings, and Alternatives (2021, background): “…layers in language models focus on local phenomena while higher layers focus on global ones (using probing classiﬁers and other methods) motivated Cao et al. (2020) to decouple question–passage processing in question-answering model, such that lower layers process the question and the passage…”
Retrieval-Free Knowledge-Grounded Dialogue Response Generation with Adapters (2021, background): “2020), by a simple decomposition in lower layers (Cao et al. 2020), or by converting a structured decoder into a non-autoregressive module (Sun et al.”
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2020, unknown): “…is most commonly used to train smaller BERT models (Sun et al., 2019; Sanh et al., 2019; Jiao et al., 2020; Zhao et al., 2019b; Turc et al., 2019; Cao et al., 2020; Sun et al., 2020b; Song et al., 2020; Mao et al., 2020; Wu et al., 2020; Li et al., 2020b; Ding and Yang, 2020; Noach and Goldberg,…”