QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension

By Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, Quoc V. Le. 2018

1,133 citations recorded by the source corpus.

Current end-to-end machine reading and question answering (Q\&A) models are primarily based on recurrent neural networks (RNNs) with attention. Despite their success, these models are often slow for both training and inference due to the sequential nature of RNNs. We propose a new Q\&A architecture called QANet, which does not require recurrent networks: Its encoder consists exclusively of convolution and self-attention, where convolution models local interactions and self-attention models global interactions. On the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 9x faster in inference, while achieving equivalent accuracy to recurrent models. The speed-up gain allows us to train the model with much more data. We hence combine our model with data generated by backtranslation from a neural machine translation model. On the SQuAD dataset, our single model, trained with augmented data, achieves 84.6 F1 score on the test set, which is significantly better than the best published F1 score of 81.8.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Model Architectures

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, methodology): “QANet is described in Yu et al. (2018), but the system has improved substantially after publication.”
Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering (2018, neutral): “…al., 2017) 39.4 50.3 ReasoNet-E (Shen et al., 2017) 39.4 49.8 Mnemonic-S (Hu et al., 2017) 46.6 56.0 Mnemonic-E (Hu et al., 2017) 46.2 55.3 QANet-S (Yu et al., 2018) 45.2 55.7 FusionNet-E (Huang et al., 2017) 51.4 60.7 SLQA-S (our) 52.1 62.7 SLQA-E (our) 54.8 64.2 SQuAD dataset (Jia and Liang,…”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “The Transformer was initially shown to be effective for machine translation, but it has subsequently been used in a wide variety of NLP settings [Radford et al., 2018; Devlin et al., 2018; McCann et al., 2018; Yu et al., 2018].”
NumNet: Machine Reading Comprehension with Numerical Reasoning (2019, neutral): “…Reader (Kadlec et al., 2016), BiDAF (Seo et al., 2017), Interactive AoA Reader (Cui et al., 2017), Gated Attention Reader (Dhingra et al., 2017), R-Net (Wang et al., 2017a), DCN (Xiong et al., 2017), QANet (Yu et al., 2018), and achieve promising results in most existing public MRC datasets.”
Technical report on Conversational Question Answering (2019, methodology): “Many MRC models have been proposed to tackle single-turn QA, such as BiDAF(Seo et al., 2016), DrQA(Chen et al., 2017), R-Net(Wang et al., 2017) and QANet(Yu et al., 2018) in SQuAD dataset.”
Unsupervised Data Augmentation for Consistency Training (2019, background): “As observed by [12], back-translation can generate diverse paraphrases while preserving the semantics of the original sentences, leading to significant performance improvements in question answering.”
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, methodology): “The hyper-parameters for our augmented QANet model are the same as the QANet baseline.”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, methodology): “QANet3 (Yu et al., 2018) is one of the top MRC models for SQuAD-style datasets.”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “For WikiSQL, this model nearly matches the prior state-of-the-art validation results of 72.4% without using a structured approach [Dong and Lapata, 2018, Huang et al., 2018, Yu et al., 2018b].”
A Simple Method for Commonsense Reasoning (2018, background): “In a broader context, LMs are used to produce good word embeddings, signiﬁcantly improved a wide variety of downstream tasks, including the general problem of question answering [20, 30].”
Improving Question Answering with External Knowledge (2019, neutral): “Compared to previous data argumentation studies for other types of QA tasks (Yu et al., 2018), differences exist in: 1) we focus on in-domain data and”
Passage Re-ranking with BERT (2019, methodology): “…MS MARCO (Nguyen et al., 2016), SearchQA (Dunn et al., 2017), TriviaQA (Joshi et al., 2017), and QUASAR-T (Dhingra et al., 2017), and the broad adoption of neural models, such as BiDAF (Seo et al., 2016), DrQA (Chen et al., 2017), DocumentQA (Clark & Gardner, 2017), and QAnet (Yu et al., 2018).”
Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering (2019, methodology): “Then the passage and question representations are input into QANet (Yu et al., 2018) to perform inter-sentence matching, and predict the ﬁnal answer.”
Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering (2019, background): “A lot of effort has been put into designing sophisticated neural MRC architectures for reading short context (e.g. a single paragraph), with much success (Wang & Jiang, 2017; Seo et al., 2017; Xiong et al., 2017; Wang et al., 2018c; Yu et al., 2018, inter alia).”
Attention Augmented Convolutional Networks (2019, methodology): “For example, the QANet [50] and Evolved Trans-”
Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension (2019, methodology): “It extends the QANet model (Yu et al., 2018) with predictions for numbers (0–9) and summation operations.”
Conformer: Convolution-augmented Transformer for Speech Recognition (2020, background): “Concurrently, papers like [15, 16] have augmented self-attention with relative position based information that maintains equivariance.”
Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, background): “However, recent breakthroughs in computer vision [3], [4], machine translation [5], [6], speech recognition [7], [8], and language understanding [9], [10] rely on large labeled datasets and make little to no use of unsupervised representation learning.”