Attention-over-Attention Neural Networks for Reading Comprehension
By Yiming Cui, Z. Chen, Si Wei, Shijin Wang, Ting-Wei Liu, Guoping Hu. 2016
455 citations recorded by the source corpus.
Cloze-style reading comprehension is a representative problem in mining relationship between document and query. In this paper, we present a simple but novel model called attention-over-attention reader for better solving cloze-style reading comprehension task. The proposed model aims to place another attention mechanism over the document-level attention and induces “attended attention” for final answer predictions. One advantage of our model is that it is simpler than related works while giving excellent performance. In addition to the primary model, we also propose an N-best re-ranking strategy to double check the validity of the candidates and further improve the performance. Experimental results show that the proposed methods significantly outperform various state-of-the-art systems by a large margin in public datasets, such as CNN and Children’s Book Test.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Caiming Xiong
- Richard Socher
- Yoshua Bengio
- Hannaneh Hajishirzi
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Jianfeng Gao
- Trevor Darrell
- Xiaogang Wang
- Cordelia Schmid
- Mohit Bansal
- Jitendra Malik
In Natural Language Processing
- Mohit Bansal
- Jianfeng Gao
- Hannaneh Hajishirzi
- Caiming Xiong
- Richard Socher
- Xipeng Qiu
- Luke S. Zettlemoyer
- Furu Wei
In Question Answering & Reading Comprehension
Citations in the corpus
- Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering (2018, methodology): “0 Interactive AoA Reader+ (Cui et al., 2016) - / - 79.”
- U-Net: Machine Reading Comprehension with Unanswerable Questions (2018, background): “…Currently, end-to-end neural network models have achieved great successes for machine reading comprehension (Seo et al. 2016; Kumar et al. 2015; Sukhbaatar et al. 2015; Cui et al. 2016; Xiong, Zhong, and Socher 2016; Dhingra et al. 2016; Shen et al. 2016; Hu et al. 2017; Wang, Yan, and Wu 2018).”
- Bidirectional Attention Flow for Machine Comprehension (2016, methodology): “Attention-over-attention model (Cui et al., 2016) uses a 2D similarity matrix between the query and context words (similar to Equation 1) to compute the weighted average of query-to-context attention.”
- NumNet: Machine Reading Comprehension with Numerical Reasoning (2019, neutral): “With the blooming of available large-scale MRC datasets, a great number of neural network-based MRC models have been proposed to answer questions for a given document including Attentive Reader (Kadlec et al., 2016), BiDAF (Seo et al., 2017), Interactive AoA Reader (Cui et al., 2017), Gated Attentio”
- Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering (2017, background): “Similarly, (Cui et al., 2017) also combined answer candidates to the question to form hypotheses, and then explicitly use language models trained on documents to re-rank the hypotheses.”
- Denoising Distantly Supervised Open-Domain Question Answering (2018, background): “Many reading comprehension systems (Chen et al., 2016; Dhin-gra et al., 2017a; Cui et al., 2017; Shen et al., 2017; Wang et al., 2017) have been proposed and achieved promising results since their multi-layer architectures and attention mechanisms allow them to reason for the question.”
- A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning (2019, neutral): “Morevoer, answer reranking has been well studied in several prior works (Cui et al., 2016; Wang et al., 2018a,b,c; Hu et al., 2019).”
- Semi-Supervised QA with Generative Domain-Adaptive Nets (2017, background): “Various neural models based on attention mechanisms (Wang and Jiang, 2016; Seo et al., 2016; Xiong et al., 2016; Wang et al., 2016; Dhingra et al., 2016; Kadlec et al., 2016; Trischler et al., 2016b; Sordoni et al., 2016; Cui et al., 2016; Chen et al., 2016) have been proposed to tackle the tasks of”
- Gated Self-Matching Networks for Reading Comprehension and Question Answering (2017, background): “End-to-end Neural Networks for Reading Comprehension Along with cloze-style datasets, several powerful deep learning models (Hermann et al., 2015; Hill et al., 2016; Chen et al., 2016; Kadlec et al., 2016; Sordoni et al., 2016; Cui et al., 2016; Trischler et al., 2016; Dhingra et al., 2016; Shen et ”
- Dynamic Coattention Networks For Question Answering (2016, methodology): “Similar to Cui et al. (2016), we also compute the summaries C Q A D of the previous attention contexts in light of each word of the document.”
- ReasoNet: Learning to Stop Reading in Machine Comprehension (2016, background): “[4] further propose to extend the query-specific attention to both query-todocument attention and document-to-query attention, which is built from the intermediate results in the query-specific attention.”
- Gated-Attention Readers for Text Comprehension (2016, background): “This detailed analysis further justifies the advantages of the GA Reader by showing its superior performance in handling questions of diverse linguistic nature.”
- Learning protein sequence embeddings using information from structure (2019, background): “Further elaborate directional alignments have been used for question answering and reading comprehension models [25, 26, 27] and for natural language inference [28, 29, 29].”
- Joint Embedding of Words and Labels for Text Classification (2018, background): “, 2016), attention over attention (Cui et al., 2016), and multi-step attention (Gehring et al.”
- Tracking the World State with Recurrent Entity Networks (2016, neutral): “However, (Trischler et al., 2016) 0.697 0.674 AoA Reader (Cui et al., 2016) 0.720 0.694 NSE Adaptive Computation (Munkhdalai & Yu, 2016) 0.732 0.714 there is still a performance gap when compared to more sophisticated machine comprehension models, many of which perform multiple layers of attention o”
- DREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension (2019, methodology): “We do not apply more recent extractive models such as AOA (Cui et al., 2017) and QANet (Yu et al., 2018) since they aim at precisely locating a span in a document.”
- MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension (2019, background): “They also attempted to substitute the MLP layer with a more complex attention-over-attention (AoA) (Cui et al., 2017) layer on top of XLNet, but did not find it to be helpful.”
- DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications (2017, background): “, 2016), AoA Reader (Cui et al., 2017), DCN (Xiong et al.”