On Evaluating the Generalization of LSTM Models in Formal Languages
By Mirac Suzgun, Yonatan Belinkov, Stuart M. Shieber. 2018
48 citations recorded by the source corpus.
Recurrent Neural Networks (RNNs) are theoretically Turing-complete and established themselves as a dominant model for language processing. Yet, there still remains an uncertainty regarding their language learning capabilities. In this paper, we empirically evaluate the inductive learning capabilities of Long Short-Term Memory networks, a popular extension of simple RNNs, to learn simple formal languages, in particular $a^nb^n$, $a^nb^nc^n$, and $a^nb^nc^nd^n$. We investigate the influence of various aspects of learning, such as training data regimes and model capacity, on the generalization to unobserved samples. We find striking differences in model performances under different training settings and highlight the need for careful analysis and assessment when making claims about the learning capabilities of neural network models.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Percy Liang
- Christopher D. Manning
- Yonatan Belinkov
- John Hewitt
- Mohit Bansal
- Yoshua Bengio
- Jonathan Berant
- Jacob Andreas
Citations in the corpus
- The EOS Decision and Length Extrapolation (2020, methodology): “These results are consistent with theoretical and empirical findings that show that LSTMs can ef-ficiently implement counting mechanisms (Weiss et al., 2018; Suzgun et al., 2019a; Merrill, 2020).”
- Theoretical Limitations of Self-Attention in Neural Sequence Models (2019, methodology): “Furthermore, infinite-precision RNNs and LSTMs can model stacks (Tabor, 2000; Grüning, 2006; Kirov and Frank, 2012) and thus are theoretically capable of modeling 2D YCK and other deterministic context-free languages perfectly.”