Phase Conductor on Multi-layered Attentions for Machine Comprehension

By R. Liu, Wei Wei, Weiguang Mao, Maria Chikina. 2017

22 citations recorded by the source corpus.

Attention models have been intensively studied to improve NLP tasks such as machine comprehension via both question-aware passage attention model and self-matching attention model. Our research proposes phase conductor (PhaseCond) for attention models in two meaningful ways. First, PhaseCond, an architecture of multi-layered attention models, consists of multiple phases each implementing a stack of attention layers producing passage representations and a stack of inner or outer fusion layers regulating the information flow. Second, we extend and improve the dot-product attention function for PhaseCond by simultaneously encoding multiple question and passage embedding layers from different perspectives. We demonstrate the effectiveness of our proposed model PhaseCond on the SQuAD dataset, showing that our model significantly outperforms both state-of-the-art single-layered and multiple-layered attention models. We deepen our results with new findings via both detailed qualitative analysis and visualized examples showing the dynamic changes through multi-layered attention models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering (2018, methodology): “In (Liu et al., 2017b) and (Shen et al., 2017), the multi-turn memory networks are designed to simulate multi-step reasoning in machine reading comprehension.”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “…hold for question answering in general, so recent models for SQuAD are not necessarily general question answering models [Yu et al., 2018a, Hu et al., 2018, Wang et al., 2017a, Liu et al., 2017b, Huang et al., 2017, Xiong et al., 2018, Liu et al., 2017a, Pan et al., 2017, Salant and Berant, 2017].”
The Effect of Natural Distribution Shift on Question Answering Models (2020, neutral): “Conductor-net (Single) (Liu et al., 2017) https://worksheets.”
Neural Approaches to Conversational AI (2018, neutral): “Single-step reasoning is simple yet efficient and the model parameters can be trained using the classical back-propagation algorithm, thus it is adopted by most of the systems (Chen et al., 2016b; Seo et al., 2016; Wang et al., 2017b; Liu et al., 2017; Chen et al., 2017a; Weissenborn et al., 2017; H”
Stochastic Answer Networks for Machine Reading Comprehension (2017, background): “Further, we also would like to explore SAN on other tasks, such as text classiﬁcation and natural language inference for its generalization in the future as well.”
Question Answering Using Hierarchical Attention on Top of BERT Features (2019, background): “In (Liu et al., 2017) and (Shen et al., 2017) the multi-turn memory networks are designed to simulate multi-step reasoning in machine reading comprehension.”
A Multi-Stage Memory Augmented Neural Network for Machine Reading Comprehension (2018, unknown): “Test (Salant and Berant, 2017) 77.58 84.16 O - QANet (Yu et al., 2018) 76.24 84.60 O O SAN (Liu et al., 2017b) 76.83 84.40 O O FusionNet (Huang et al., 2017b) 75.97 83.90 O O RaSoR + TR (Salant and Berant, 2017) 75.79 83.26 O - Conducter-net (Liu et al., 2017a) 74.41 82.74 O O Reinforced Mnemonic Re”
Making Neural Machine Reading Comprehension Faster (2019, methodology): “Most of the papers that approach to solve SQuAD problem have utilized RNN based models [3, 5, 7, 9, 13, 14, 16, 17, 20, 22–24, 26, 29–31, 34–39, 41, 43] and they follow a similar chain of processes.”
Weaver: Deep Co-Encoding of Questions and Documents for Machine Reading (2018, methodology): “In this paper, we instead propose to co-encode questions and contexts with the same recurrent layers directly and use an answering step inspired by the multi-hops approach used in (Sukhbaatar et al., 2015; Liu et al., 2017b).”
PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text (2019, background): “A long line of QA models have been developed which answer questions based on a single passage of text (Dhingra et al., 2016; Yu et al., 2018; Seo et al., 2016; Gao et al., 2018; Liu et al., 2017; De-vlin et al., 2018).”