Densely Connected Attention Propagation for Reading Comprehension

By Yi Tay, Anh Tuan Luu, Siu Cheung Hui, Jian Su. 2018

48 citations recorded by the source corpus.

We propose DecaProp (Densely Connected Attention Propagation), a new densely connected neural architecture for reading comprehension (RC). There are two distinct characteristics of our model. Firstly, our model densely connects all pairwise layers of the network, modeling relationships between passage and query across all hierarchical levels. Secondly, the dense connectors in our network are learned via attention instead of standard residual skip-connectors. To this end, we propose novel Bidirectional Attention Connectors (BAC) for efficiently forging connections throughout the network. We conduct extensive experiments on four challenging RC benchmarks. Our proposed approach achieves state-of-the-art results on all four, outperforming existing baselines by up to 2.6% to 14.2% in absolute F1 score.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

A Discrete Hard EM Approach for Weakly Supervised Question Answering (2019, background): “…variable during learning, but most prior work on reading comprehension has rather focused on the model architecture and used heuristics to map the weak signal to full supervision (e.g. by selecting the ﬁrst answer span in T RIVIA QA (Joshi et al., 2017; Tay et al., 2018; Talmor and Berant, 2019)).”
NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, methodology): “Residual connections [16] and dense connections [19] were adopted from computer vision architectures and impacted on machine translation [7], text classification [43], and machine reading comprehension [40].”
MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension (2019, neutral): “1 M ULTI QA improves state-of-the-art performance on ﬁvedatasets, although it does not even 1 State-of-the-are-results were found in (Tay et al., 2018) for N EWS QA, in Lin et al. (2018), for S EARCH QA, in Das et al. (2019) for TQA-U, in (Talmor and Berant, 2018b) for CWQ, in Ding et al. (2019)…”
Select, Answer and Explain: Interpretable Multi-hop Reading Comprehension over Multiple Documents (2019, background): “Numerous neural models have been proposed (Seo et al. 2016; Xiong, Zhong, and Socher 2016; Tay et al. 2018) and achieved promising performances on several different MRC data sets, such as SQuAD (Rajpurkar et al. 2016; Rajpurkar, Jia, and Liang 2018), NarrativeQA (Koˇcisk`y et al. 2018) and CoQA…”
OmniNet: Omnidirectional Representations from Transformers (2021, background): “(Tay et al., 2018) proposed to densely connect the attention across stacked RNN encoder layers for language understanding tasks.”
Rethinking Search: Making Experts out of Dilettantes (2021, background): “Likewise, a surge of neural networkbased models, primarily attention-based [86, 97, 105], have been developed for tackling this question answering task variant.”
Multi-style Generative Reading Comprehension (2019, methodology): “We used the reports of top-performing extractive (Seo et al., 2017; Tay et al., 2018; Hu et al., 2018) and generative (Bauer et al.”
CogLTX: Applying BERT to Long Texts (2020, methodology): “The SOTA model HGN [14] leverages extra hyperlink data in Wikipedia, based on which the dataset is constructed.”
Retrospective Reader for Machine Reading Comprehension (2020, unknown): “9.6 34.1 48.2 mLSTM (Wang and Jiang 2017) 34.4 49.6 34.9 50.0 BiDAF (Seo et al. 2016) - - 37.1 52.3 R2-BiLSTM (Weissenborn 2017) - - 43.7 56.7 AMANDA (Kundu and Ng 2018) 48.4 63.3 48.4 63.7 DECAPROP (Tay et al. 2018) 52.5 65.7 53.1 66.3 BERT (Devlin et al. 2019) - - 46.5 56.7 NeurQuRI (Back et al. 2”
Explicit Utilization of General Knowledge in Machine Reading Comprehension (2018, background): “attention as a skip-connector to densely connect pairwise layers (Tay et al., 2018).”