LSTM recurrent networks learn simple context-free and context-sensitive languages

By Felix Alexander Gers, J. Urgen Schmidhuber. 2001

760 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

A Convolutional Neural Network for Modelling Sentences (2014, methodology): “On the hand-labelled test set, the network achieves a greater than 25% reduction in the prediction error with respect to the strongest unigram and bigram baseline reported in Go et al. (2009).”
Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (2015, background): “k (LSTM) architecture. While this model was orginally developed to address the vanishing and exploding gradient problems, LSTM is also able to learn simple context-free and context-sensitive grammars [17, 37]. This is possible because its hidden units can choose through a multiplicative gating mecha”
ShapeWorld - A new test methodology for multimodal language understanding (2017, methodology): “A S HAPE W ORLD dataset differs from standard VQA evaluation datasets in three main ways.”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “A long tradition in work on neural networks is to evaluate and analyze their ability to learn different formal languages (Das et al., 1992; Casey, 1996; Gers and Schmidhuber, 2001; Bodén and Wiles, 2002; Chalup and Blair, 2003).”
Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure (2017, neutral): “…neural networks are known to be Turing complete (Siegelmann and Sontag, 1995) and many papers can be found that demonstrate the capability of (hand-crafted) recurrent networks to implement small context-free or even context sensitive languages (e.g. Gers and Schmidhuber, 2001; Rodriguez, 2001).”
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies (2016, positive): “Previous research has shown that RNNs (in particular LSTMs) can learn artificial context-free languages (Gers and Schmidhuber, 2001) as well as nesting and”
Colorless Green Recurrent Networks Dream Hierarchically (2018, background): “and even context-sensitive grammars (e.g., Elman, 1991, 1993; Rohde and Plaut, 1997; Christiansen and Chater, 1999; Gers and Schmidhuber, 2001; Cartling, 2008).”
Deep learning in neural networks: An overview (2014, background): “CECs are themain reason why LSTM nets can learn to discover the importance of (andmemorize) events that happened thousands of discrete time steps ago,while previous RNNs already failed in case of minimal time lags of 10 steps.”
Learning Precise Timing with LSTM Recurrent Networks (2003, background): “…and Fallside, 1987, Williams and Zipser, 1992), or combinations thereof (Schmidhuber, 1992, Williams and Zipser, 1992), already fail to learn in the presence of 10-step time lags (Hochreiter, 1991, Bengio et al., 1994, Hochreiter and Schmidhuber, 1997, Gers et al., 2000, Hochreiter et al., 2001).”
Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks (2016, positive): “This observation simply conﬁrms earlier observations that has demonstrate simple context-free models in RNNs and LSTMs [7,25].”
Self-Delimiting Neural Networks (2012, background): “Long Short-Term Memory (LSTM) overcame these problems, outperforming early RNNs in many applications [27, 14, 15, 65, 20, 21, 23, 22].”
Theoretical Limitations of Self-Attention in Neural Sequence Models (2019, background): “A range of studies has experimentally examined the ability of recurrent networks to model counter languages such as anbn (Kalinke and Lehmann, 1998; Gers and Schmidhuber, 2001; Cartling, 2008; Weiss et al., 2018; Suzgun et al., 2019).”
Synthetic Data for Deep Learning (2019, background): “For example, Joulin and Mikolov [294] present a new architecture (stack-augmented recurrent networks) to learn regularities in algorithmically generated sequences of symbols; the training data, as in previous such works [200,247,251,637], is synthetically generated by hand-crafted simple algorithms,”
Deep Learning for an Effective Nonorthogonal Multiple Access Scheme (2018, background): “LSTM is a special type of RNN architecture that is capable of remembering values over arbitrary intervals [28].”
Learning to Recombine and Resample Data for Compositional Generalization (2020, background): “In contrast, recurrent neural networks can accurately model a n b n (Weiss et al., 2018; Gers & Schmidhuber, 2001).”
On Evaluating the Generalization of LSTM Models in Formal Languages (2018, background): “Most of the previous studies trained networks on sequences of lengths n [1, N ], where typical N values were between 10 and 50 (Bodén and Wiles, 2000; Gers and Schmidhuber, 2001), and more recently 100 (Weiss et al.”
Neural Arithmetic Logic Units (2018, background): “The question of whether and how recurrent networks generalize to sequences longer than they encountered in training has been of enduring interest, especially since well-formed sentences in human languages are apparently unbounded in length, but are learned from a limited sample [9, 19, 28].”