Long Short-Term Memory-Networks for Machine Reading

By Jianpeng Cheng, Li Dong, Mirella Lapata. 2016

1,177 citations recorded by the source corpus.

In this paper we address the question of how to render sequence-level networks better at handling structured input. We propose a machine reading simulator which processes text incrementally from left to right and performs shallow reasoning with memory and attention. The reader extends the Long Short-Term Memory architecture with a memory network in place of a single memory cell. This enables adaptive memory usage during recurrence with neural attention, offering a way to weakly induce relations among tokens. The system is initially designed to process a single sequence but we also demonstrate how to integrate it with an encoder-decoder architecture. Experiments on language modeling, sentiment analysis, and natural language inference show that our model matches or outperforms the state of the art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Attention is All you Need (2017, background): “Self-attention has been used successfully in a variety of tasks including reading comprehension, abstractive summarization, textual entailment and learning task-independent sentence representations [4, 22, 23, 19].”
A Structured Self-attentive Sentence Embedding (2017, methodology): “The LSTMN model (Cheng et al., 2016) also proposed a very successful intra-sentence level attention mechanism, which is later used by Parikh et al. (2016).”
Simple and Effective Multi-Paragraph Reading Comprehension (2017, methodology): “Self-attention has been used in several prior works (Cheng et al., 2016; Wang et al., 2017b; Pan et al., 2017).”
A Decomposable Attention Model for Natural Language Inference (2016, methodology): “However, we can augment this input representation with intra-sentence attention to encode compositional relationships between words within each sentence, as proposed by Cheng et al. (2016).”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “The primary building block of the Transformer is self-attention [Cheng et al., 2016].”
Graph Attention Networks (2017, background): “Together with Recurrent Neural Networks (RNNs) or convolutions, self-attention has proven to be useful for tasks such as machine reading (Cheng et al., 2016) and learning sentence representations (Lin et al., 2017).”
On Measuring and Mitigating Biased Inferences of Word Embeddings (2019, methodology): “Our NLI models for GloVe and ELMo are based on the decomposable attention model (Parikh et al. 2016) with a BiLSTM encoder instead of the original projective one (Cheng, Dong, and Lapata 2016).”
CCNet: Criss-Cross Attention for Semantic Segmentation (2018, background): “Non-local Networks [32] utilizes a self-attention mechanism [10, 29], which enable a single feature from any position to perceive features of all the other positions, leading to generate more power pixel-wise representation.”
Image Transformer (2018, background): “In this work we show that self-attention (Cheng et al., 2016; Parikh et al., 2016; Vaswani et al., 2017) can achieve a better balance in the trade-off between the virtually unlimited receptive field of the necessarily sequential PixelRNN and the limited receptive field of the much more parallelizabl”
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022, background): “The main architectural unit of the Transformer is a Transformer block , which consists of (at minimum) multi-headed self attention [Cheng et al., 2016], layer normalization [Ba et al., 2016], a dense two-layer feedforward network, and residual connections [He et al., 2016].”
Self-Attention Generative Adversarial Networks (2018, background): “Self-attention [4, 20, 32], on the other hand, exhibits a better balance between ability to model long-range dependencies and computational and statistical efﬁciency.”
AllenNLP: A Deep Semantic Natural Language Processing Platform (2018, background): “There is a large number of ways to do this, including LSTMs (Hochreiter and Schmidhuber, 1997), GRUs ( Cho et al., 2014), intra-sentence attention (Cheng et al., 2016), recurrent additive networks ( Lee et al., 2017b), and many more.”
Gated Self-Matching Networks for Reading Comprehension and Question Answering (2017, background): “Cheng et al. (2016) propose a novel LSTM network to encode words in a sentence which considers the relation between the current token being processed and its past tokens in the memory.”
Enhanced LSTM for Natural Language Inference (2016, background): “…attention (Rocktäschel et al., 2015) 250K 85.3 83.5 (9) 300D mLSTM (Wang and Jiang, 2016) 1.9M 92.0 86.1 (10) 450D LSTMN with deep attention fusion (Cheng et al., 2016) 3.4M 88.5 86.3 (11) 200D decomposable attention model (Parikh et al., 2016) 380K 89.5 86.3 (12) Intra-sentence attention + (11)…”
Are Sixteen Heads Really Better than One? (2019, background): “Attention was shortly adapted (successfully) to other NLP tasks, often achieving then state-of-the-art performance in reading comprehension (Cheng et al., 2016), natural language inference (Parikh et al., 2016) or abstractive summarization (Paulus et al., 2017) to cite a few.”
Rationalizing Neural Predictions (2016, background): “Attention based models offer another lens to the inner workings of neural models (Bahdanau et al., 2015; Cheng et al., 2016; Martins and Astudillo, 2016; Chen et al., 2015; Xu and Saenko, 2015; Yang et al., 2015).”
Bottom-Up Abstractive Summarization (2018, positive): “Proposed solutions include modifying models with extensions such as a coverage mechanism (Tu et al., 2016; See et al., 2017) or intra-sentence attention (Cheng et al., 2016; Paulus et al., 2017).”
Do Transformer Modifications Transfer Across Implementations and Applications? (2021, methodology): “Each layer block in the encoder comprises a multi-headed self-attention mechanism (Cheng et al., 2016) followed by a position-wise dense/nonlinearity/dense feedforward network.”