Neural Semantic Encoders

By Tsendsuren Munkhdalai, Hong Yu. 2016

140 citations recorded by the source corpus.

We present a memory augmented neural network for natural language understanding: Neural Semantic Encoders. NSE is equipped with a novel memory update rule and has a variable sized encoding memory that evolves over time and maintains the understanding of input sequences through read, compose and write operations. NSE can also access 1 multiple and shared memories. In this paper, we demonstrated the effectiveness and the flexibility of NSE on five different natural language tasks: natural language inference, question answering, sentence classification, document sentiment analysis and machine translation where NSE achieved state-of-the-art performance when evaluated on publically available benchmarks. For example, our shared-memory model showed an encouraging result on neural machine translation, improving an attention-based baseline by approximately 1.0 BLEU.

Subfields: Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

A Structured Self-attentive Sentence Embedding (2017, methodology): “…2016b) 83.3% 300D Tree-based CNN encoders (Mou et al., 2015a) 82.1% 300D SPINN-PI encoders (Bowman et al., 2016) 83.2% 300D NTI-SLSTM-LSTM encoders (Munkhdalai & Yu, 2016a) 83.4% 1024D GRU encoders with SkipThoughts pre-training (Vendrov et al., 2015) 81.4% 300D NSE encoders (Munkhdalai & Yu,…”
Learned in Translation: Contextualized Word Vectors (2017, neutral): “…in NLP. [Looks et al., 2017] 89.4 SVM [Van-Tu and Anh-Cuong, 2016] 95.2 TE-LSTM [Huang et al., 2017] 89.6 DSCNN-P [Zhang et al., 2016] 95.6 NSE [Munkhdalai and Yu, 2016a] 89.7 BCN+Char+CoVe (Ours) 95.8 BCN+Char+CoVe (Ours) 90.3 TBCNN [Mou et al., 2015] 96.0 bmLSTM [Radford et al., 2017] 91.8…”
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference (2017, background): “…for sentence understanding and spurring work on core representation learning techniques for NLU such as attention (Wang and Jiang, 2016), memory (Munkhdalai and Yu, 2017), and the use of parse structure (Mou et al., 2016b; The 8 million dollars for emergency housing was still not enough to…”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “Sentiment Analysis Because SST came with parse trees for every example, some approaches use all of the sub-tree labels by modeling trees explicitly [Yu and Munkhdalai, 2017b, Tai et al., 2015] as in the original paper.”
Enhanced LSTM for Natural Language Inference (2016, methodology): “The model of Munkhdalai and Yu (2016b) extends the framework of Wang and Jiang (2016) to a full n-ary tree model and achieves further improvement.”
Gated-Attention Readers for Text Comprehension (2016, background): “This detailed analysis further justifies the advantages of the GA Reader by showing its superior performance in handling questions of diverse linguistic nature.”
Natural Language Inference over Interaction Space (2017, methodology): “…representation for each sentence and classiﬁes the relation by using the concatenation of two vector representation along with their absolute element-wise difference and element-wise product(Bowman et al., 2016; Vendrov et al., 2015; Mou et al., 2015; Liu et al., 2016; Munkhdalai & Yu, 2016).”
Meta Networks (2017, background): “MANNs have shown promising re-sults on a range of tasks starting from small programming problems (Graves et al., 2014) to large-scale language tasks (Weston et al., 2015; Munkhdalai & Yu, 2017).”
On the Binding Problem in Artificial Neural Networks (2020, background): “…addressing operations, e.g. based on the order in which memory locations are accessed (DNC; Graves et al., 2016), based on when they were last read (Munkhdalai and Yu, 2017), or based on a key-value addressing scheme (Csordás and Schmidhuber, 2019) may confer additional generalization capabilities…”
Recurrent Neural Networks for Classifying Relations in Clinical Notes (2017, neutral): “In general domain NLP, a growing number of studies have successfully used recurrent neural networks (RNNs) combined with word embedding [22] on tasks including language modeling [23], text classification [24–27], question answering [25,26,28,29], machine translation [25,30–32], named entity recognit”
Shortcut-Stacked Sentence Encoders for Multi-Domain Inference (2017, neutral): “We use cross-entropy loss as the training objective with Adam-based (Kingma and Ba, 2014) opti-Model Accuracy SNLI Multi-NLI Matched Multi-NLI Mismatched CBOW (Williams et al., 2017) 80.6 65.2 64.6 biLSTM Encoder (Williams et al., 2017) 81.5 67.5 67.1 300D Tree-CNN Encoder (Mou et al., 2015) 82.1 – ”
Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations (2019, background): “The SNLI corpus [3] of ∼ 570 K such human-written instances enabled a plethora of works on this task [28, 25, 22].”
DiSAN: Directional Self-Attention Network for RNN/CNN-free Language Understanding (2017, neutral): “ree-based CNN encoders (Mou et al. 2016) 3.5m 83.3 82.1 300D SPINN-PI encoders (Bowman et al. 2016) 3.7m 89.2 83.2 600D Bi-LSTM encoders (Liu et al. 2016) 2.0m 86.4 83.3 300D NTI-SLSTM-LSTM encoders (Munkhdalai and Yu 2017b) 4.0m 82.5 83.4 600D Bi-LSTM encoders+intra-attention (Liu et al. 2016) 2.8m”
Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling (2018, neutral): “…89.2 83.2 600D Bi-LSTM encoders (Liu et al., 2016) 2.0m 86.4 83.3 300D NTI-SLSTM-LSTM encoders (Munkhdalai & Yu, 2017b) 4.0m 82.5 83.4 600D Bi-LSTM encoders+intra-attention (Liu et al., 2016) 2.8m 84.5 84.2 300D NSE encoders (Munkhdalai & Yu, 2017a) 3.0m 86.2 84.6 600D (300+300) Deep Gated Attn.”
DR-BiLSTM: Dependent Reading Bidirectional LSTM for Natural Language Inference (2018, background): “Various deep learning models have been proposed that achieve successful results for this task (Gong et al., 2017; Wang et al., 2017; Chen et al., 2017; Yu and Munkhdalai, 2017a; Parikh et al., 2016; Zhao et al., 2016; Sha et al., 2016).”
Natural Language Inference with External Knowledge (2017, methodology): “…models, which often need relatively large amounts of annotated data to estimate their parameters, have shown to achieve the state of the art on SNLI and MultiNLI (Bowman et al., 2015; 2016; Munkhdalai & Yu, 2016b; Parikh et al., 2016; Sha et al., 2016; Paria et al., 2016; Chen et al., 2017a;b).”
A Compare-Propagate Architecture with Alignment Factorization for Natural Language Inference (2017, result): “Though we compare with many other models (Wang and Jiang, 2016b; Rocktäschel et al., 2015; Parikh et al., 2016; Yu and Munkhdalai, 2017; Sha et al., 2016), the key SoTA competitors on this dataset are the BiMPM (Wang et al.”