Attention-Guided Answer Distillation for Machine Reading Comprehension

By Minghao Hu, Yuxing Peng, Furu Wei, Zhen Huang, Dongsheng Li, Nan Yang, M. Zhou. 2018

76 citations recorded by the source corpus.

Despite that current reading comprehension systems have achieved significant advancements, their promising performances are often obtained at the cost of making an ensemble of numerous models. Besides, existing approaches are also vulnerable to adversarial attacks. This paper tackles these problems by leveraging knowledge distillation, which aims to transfer knowledge from an ensemble model to a single model. We first demonstrate that vanilla knowledge distillation applied to answer span prediction is effective for reading comprehension systems. We then propose two novel approaches that not only penalize the prediction on confusing answers but also guide the training with alignment information distilled from the ensemble. Experiments show that our best student model has only a slight drop of 0.4% F1 on the SQuAD test set compared to the ensemble teacher, while running 12x faster during inference. It even outperforms the teacher on adversarial SQuAD datasets and NarrativeQA benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

TinyBERT: Distilling BERT for Natural Language Understanding (2019, methodology): “C L ] 2 4 Se p 20 19 KD has been extensively studied in NLP (Kim & Rush, 2016; Hu et al., 2018), while designing KD methods for BERT has been less explored.”
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, background): “Knowledge of the attention maps (Zagoruyko & Komodakis, 2017; Hu et al., 2018) is also introduced to help the training.”
Knowledge Distillation: A Survey (2020, methodology): “…for solving the numerous NLP tasks (Liu et al. 2019b; Gordon and Duh 2019; Haidar and Rezagholizadeh 2019; Yang et al. 2020c; Tang et al. 2019; Hu et al. 2018; Sun et al. 2019; Nakashole and Flauger 2017; Jiao et al. 2020; Wang et al. 2018d; Zhou et al. 2019a; Sanh et al. 2019; Turc et al.…”
Well-Read Students Learn Better: The Impact of Student Initialization on Knowledge Distillation (2019, methodology): “Surprisingly, student model initialization has received relatively little attention while a substantial body of work has focused on speciﬁc student architectures and novel transfer methods (Ba & Caruana, 2014; Romero et al., 2014; Kim & Rush, 2016; Yim et al., 2017; Hu et al., 2018).”
MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension (2019, background): “M ULTI QA obtained 66.7 EM and 73.1 F 1 , outperforming BERT-large (60.4EM, 66.3F1) by a signiﬁcant margin, and also substantially improving state-of-the-art re-sults (56.0 EM, 61.3 F 1 , (Hu et al., 2018) and 52.1 EM, 62.7 F 1 , (Wang et al., 2018)).”
Attention in Natural Language Processing (2019, background): “This is what happens, for example, in attention distillation, where a student network is trained to penalize the most confusing features according to a teacher network, producing an efficient and robust model in the task of machine reading comprehension [155].”
Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals (2021, unknown): “[11] applied the teacher-student framework to visual question answering task and reading comprehension task, respectively.”
Densely Connected Attention Propagation for Reading Comprehension (2018, methodology): “(5)Notably, a new SOTA was set by [Hu et al., 2018] after the NIPS submission deadline.”
Supplementary Material: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, background): “Knowledge of the attention maps [42, 11] is also introduced to help the training.”
Multi-style Generative Reading Comprehension (2019, methodology): “Hu et al. (2018) proposed an answer verifier to compare the answer sentence with the question.”
Knowledge Squeezed Adversarial Network Compression (2019, background): “Attention transfer is also introduced in machine reading comprehension [45] to transfer teacher attentive information.”
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers (2020, background): “More ﬁne-grained knowledge such as hidden states (Romero et al., 2015) and attention distributions (Zagoruyko and Komodakis, 2017; Hu et al., 2018) are introduced to improve the student model.”
KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization (2021, background): “For natural language processing tasks, KD has been extensively studied Hu et al. (2018), Jiao et al. (2019).”
Knowledge Distillation in Document Retrieval (2019, methodology): “(Hu et al. (2018)) have used an ensemble of models as the teacher model, similar to (Hinton et al. (2015)), to guide the alignments of the student model in machine reading comprehension.”
Efficient Video Classification Using Fewer Frames (2019, background): “This idea of Knowledge Distillation has also been tried in the context of pruning networks for multiple object detection [4], speech recognition [38] and reading comprehension [14].”
On Knowledge distillation from complex networks for response prediction (2019, background): “In the context of reading comprehension or span prediction, Hu et al. (2018) have very recently shown that we can distill knowledge from an ensemble of models into a single model.”