MEMEN: Multi-layer Embedding with Memory Networks for Machine Comprehension

By Boyuan Pan, Hao Li, Zhou Zhao, Bin Cao, Deng Cai, Xiaofei He. 2017

84 citations recorded by the source corpus.

Machine comprehension(MC) style question answering is a representative problem in natural language processing. Previous methods rarely spend time on the improvement of encoding layer, especially the embedding of syntactic information and name entity of the words, which are very crucial to the quality of encoding. Moreover, existing attention methods represent each query word as a vector or use a single vector to represent the whole query sentence, neither of them can handle the proper weight of the key words in query sentence. In this paper, we introduce a novel neural network architecture called Multi-layer Embedding with Memory Network(MEMEN) for machine reading task. In the encoding layer, we employ classic skip-gram model to the syntactic and semantic information of the words to train a new kind of embedding layer. We also propose a memory network of full-orientation matching of the query and passage to catch more pivotal information. Experiments show that our model has competitive results both from the perspectives of precision and efficiency in Stanford Question Answering Dataset(SQuAD) among all published results and achieves the state-of-the-art results on TriviaQA dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, background): “Zhou et al. (2017) improved the diversity of the SQuAD data by generating more questions.”
Simple and Effective Multi-Paragraph Reading Comprehension (2017, methodology): “All Veriﬁed EM F1 EM F1 baseline (Joshi et al., 2017) 40.74 47.06 49.54 55.80 MEMEN* (Pan et al., 2017) 43.16 46.90 49.28 55.83 Mnemonic Reader (Hu et al., 2017) 46.94 52.85 54.45 59.46 Reading Twice for NLU (Weissenborn et al., 2017a) approaches as the model is used to independently process an…”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “This quirk of the dataset does not hold for question answering in general, so recent models for SQuAD are not necessarily general question answering models (Yu et al., 2018a; Hu et al., 2018; Wang et al., 2017a; Liu et al., 2017b; Huang et al., 2017; Xiong et al., 2018; Liu et al., 2017a; Pan et al.”
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (2018, methodology): “petitive architectures (Liu et al., 2018; Clark and Gardner, 2017; Wang et al., 2017; Seo et al., 2017; Pan et al., 2017; Salant and Berant, 2018; Xiong et al., 2018) on SQuAD can be viewed as similar to our implemented model.”
The Effect of Natural Distribution Shift on Question Answering Models (2020, background): “MEMEN (Ensemble; original model in paper) (Pan et al., 2017) https://worksheets.”
Multi-Passage Machine Reading Comprehension with Cross-Passage Answer Verification (2018, background): “Great efforts have also been made to develop models for these MRC tasks , especially for the answer extraction on single passage (Wang and Jiang, 2016; Seo et al., 2016; Pan et al., 2017).”
HAS-QA: Hierarchical Answer Spans Model for Open-domain Question Answering (2019, background): “2017; Weissenborn, Wiese, and Seiffe 2017), the others use the pointer networks (Wang and Jiang 2016; Seo et al. 2016; Wang et al. 2017; Pan et al. 2017).”
Multi-grained Attention Network for Aspect-Level Sentiment Classification (2018, background): “In contrast, motivated by the bidirectional attention ﬂow approaches (Seo et al., 2017; Pan et al., 2017) in machine comprehension, we propose a ﬁne-grained attention mechanism which is responsible for linking and fusing information from the aspect and the context words.”
A Multi-Stage Memory Augmented Neural Network for Machine Reading Comprehension (2018, background): “O MEMEN (Pan et al., 2017) 70 5.”
Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks (2017, background): “Recently, it has demonstrated the state-of-the-art performance in numerous NLP applications, including the reading comprehension [20] and textual question answering [6, 16].”