Context-free and context-sensitive dynamics in recurrent neural networks

By Mikael Bodén, Janet Wiles. 2000

81 citations recorded by the source corpus.

Continuous-valued recurrent neural networks can learn mechanisms for processing context-free languages. The dynamics of such networks is usually based on damped oscillation around fixed points in state space and requires that the dynamical components are arranged in certain ways. It is shown that qualitatively similar dynamics with similar constraints hold for anbncn , a context-sensitive language. The additional difficulty with anbncn , compared with the context-free language anbn , consists of 'counting up' and 'counting down' letters simultaneously. The network solution is to oscillate in two principal dimensions, one for counting up and one for counting down. This study focuses on the dynamics employed by the sequential cascaded network, in contrast to the simple recurrent network, and the use of backpropagation through time. Found solutions generalize well beyond training data, however, learning is not reliable. The contribution of this study lies in demonstrating how the dynamics in recurrent neural networks that process context-free languages can also be employed in processing some context-sensitive languages (traditionally thought of as requiring additional computation resources). This continuity of mechanism between language classes contributes to our understanding of neural networks in modelling language learning and processing.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Model Architectures

Citations in the corpus

Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (2015, methodology): “At the same time, our model seems simpler than many other models used for this type of tasks (Tabor, 2000; Bod´en & Wiles, 2000; Graves et al., 2014).”
Deep learning in neural networks: An overview (2014, neutral): “Those that did not (Bodén & Wiles, 2000; Rodriguez et al., 1999) failed to extract the general rules, and did not generalizewell on substantially larger test sets.”
On Evaluating the Generalization of LSTM Models in Formal Languages (2018, background): “…studies have examined various length distribution models to generate appropriate training sets for each formal language: Wiles and Elman (1995); Bod´en and Wiles (2000); Ro-driguez (2001), for instance, used length distributions that were skewed towards having more short sequences than long…”
Incremental training of first order recurrent neural networks to predict a context-sensitive language (2003, methodology): “Bodén and Wiles (2000, 2001) did not use the end-of-string marker and updated the weights at the end of each string without resetting at all.”