Learning to count without a counter: A case study of dynamics and activation landscapes in recurrent networks

By Jeffrey L. Elman, Janet Wiles. 1995

Subfields: Representation & Self-Supervised Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Model Architectures

Citations in the corpus

Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (2015, methodology): “Wiles & Elman (1995) has shown that it was possible to learn a standard recurrent network able to count on a limited range.”
Synthetic Data for Deep Learning (2019, background): “For example, Joulin and Mikolov [294] present a new architecture (stack-augmented recurrent networks) to learn regularities in algorithmically generated sequences of symbols; the training data, as in previous such works [200,247,251,637], is synthetically generated by hand-crafted simple algorithms,”
Evolving Memory Cell Structures for Sequence Learning (2009, background): “10 [17], while LSTM have been shown to generalize up to over n = 100 [2].”
Computation in Recurrent Neural Networks: From Counters to Iterated Function Systems (1998, unknown): “3 Setting up a Counter By using linear saturation instead of sigmoidal activation functions it is possible to avoid some di culties occurring in the design of a RNN that implements a counter as described and discussed in [4] and [7].”
Context-free and context-sensitive dynamics in recurrent neural networks (2000, background): “The SRN has been shown to use qualitatively di erent dynamics from the nite state automata emulations, which allow it to generalize when learning context-free languages (Wiles and Elman, 1995; Rodriguez and Wiles, 1998; Rodriguez et al., 1999; Bod en et al., 1999).”
Fractal encoding of context‐free grammars in connectionist networks (2000, methodology): “: Rodriguez et al. (1999) Wiles and Elman (1995) study a backpropagation network that is trained on samples from the closely related language l n r n , n {1, 2, 3, % }.”
The Fine Line between Linguistic Generalization and Failure in Seq2Seq-Attention Models (2018, background): “Further work explored how RNNs can learn context free languages (Wiles and Elman, 1995; Ro-driguez and Wiles, 1998) as well some context sensitive languages (Gers and Schmidhuber, 2001).”
Incremental training of first order recurrent neural networks to predict a context-sensitive language (2003, methodology): “Wiles and Elman (1995) extended these approaches to non-regular languages and showed how SRNs can be trained by backpropagation through time (BPTT) (Rumelhart et al., 1986; Zipser, 1990) to predict symbols in finite string sequences of the context-free language {a n b n ; n $ 1}.”
Designing a Counter: Another Case Study of Dynamics and Activation Landscapes in Recurrent Networks (1997, methodology): “After extensivetraining, the networks of Wiles and Elman performed poorly, most of themrecognized elements of a b , which is a much simpler task not requiring a counterat all, whereas only a single network was able to generalize to strings of lengthn = 21 , and after \making additional small adjustm”
Kalman filters improve LSTM network performance in problems unsolvable by traditional recurrent nets (2003, background): “Only few authors have tried to teach RNNs to extract the rules of simple context free and context sensitive languages (CFLs and CSLs) whose recognition requires the functional equivalent of a potentially unlimited stack (Sun et al., 1993; Wiles and Elman, 1995; Boden and Wiles, 2000).”
Rule Extraction from Recurrent Neural Networks: ATaxonomy and Review (2005, background): “…& Tanenhaus 1999), activation values plotted over time (e.g. Husbands, Harvey & Cliﬀ 1995, Meeden 1996, Ziemke & Thieme 2002), iterated maps (e.g. Wiles & Elman 1995), vector ﬂow ﬁelds (e.g. Rodriguez et al. 1999, Rodriguez 1999), external behaviour analysis of RNN-controlled autonomous…”
Learning a context-free task with a recurrent neural network: An analysis of stability (1999, background): “Wiles and Elman (1995) reported that training a network after it had found a solution often resulted in the network losing the solution.”
A Recurrent Neural Network that Learns to Count (1999, methodology): “Network 3 has dynamics similar to network 1.10 Network 4 has dynamics similar to network 2, which was reported earlier by Wiles and Elman (1995).”
A Recurrent Network that performs a Context-Sensitive Prediction Task (1996, methodology): “Wiles & Elman (1995) studied the behavior of a RN on the language anbn which is context-free but not regular.”
On Evaluating the Generalization of LSTM Models in Formal Languages (2018, result): “Previous studies have examined various length distribution models to generate appropriate training sets for each formal language: Wiles and Elman (1995); Bod´en and Wiles (2000); Ro-driguez (2001), for instance, used length distributions that were skewed towards having more short sequences than…”