Recurrent Neural Networks Can Learn to Implement Symbol-Sensitive Counting

By P. Rodríguez, Janet Wiles. 1997

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, methodology): “…free languages (CFLs) that cannot be represented by HMMs or similar FSAs discussed in the RNN literature (Andrews, Diederich, & Tickle, 1995; Rodriguez & Wiles, 1998; Rodriguez, Wiles, & Elman, 1999; Steijvers & Grunwald, 1996; Sun, Giles, Chen, & Lee, 1993; Tonkes & Wiles, 1997; Wiles &…”
Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks (2016, positive): “This observation simply conﬁrms earlier observations that has demonstrate simple context-free models in RNNs and LSTMs [7,25].”
LSTM recurrent networks learn simple context-free and context-sensitive languages (2001, methodology): “Unlike the previous approach of [ 16 ], LSTM easily learns the complete training set and reliably finds solutions that generalize well.”
Compositional Generalization for Primitive Substitutions (2019, background): “There has been exploration on compositionality in neural networks for systematic behaviour (Wong and Wang, 2007; Brakel and Frank, 2009), counting ability (Rodriguez and Wiles, 1998; Weiss et al., 2018) and sensitivity to hierarchical structure (Linzen et al., 2016).”
The Fine Line between Linguistic Generalization and Failure in Seq2Seq-Attention Models (2018, methodology): “As done in previous work (Lake and Baroni, 2018; Rodriguez and Wiles, 1998), we ease analysis by looking at a simple formal task.”
Computation in Recurrent Neural Networks: From Counters to Iterated Function Systems (1998, unknown): “in [4] and [12] simple RNNs were trained to learn context free languages up to a certain word length.”
Kalman filters improve LSTM network performance in problems unsolvable by traditional recurrent nets (2003, background): “Some previous RNNs even failed to learn small CFL training sets (Rodriguez and Wiles, 1998).”
A Recurrent Neural Network that Learns to Count (1999, background): “We have results showing that an RNN can learn the task with some generalization such that an idealized version of the network dynamics re¯ ects a counting solution (Rodriguez & Wiles, 1998).”
Evolving Memory Cell Structures for Sequence Learning (2009, background): “LSTM, unlike conventional RNNs, has been shown to be able to capture longterm time dependencies, learn precise timing, and generalize well on examples of both context-free and context-sensitive languages such as ab and abc, respectively, whereas normal RNNs completely failed to capture the underlyin”
Context-free and context-sensitive dynamics in recurrent neural networks (2000, background): “The SRN has been shown to use qualitatively diﬀerent dynamics from the ﬁnite state automata emulations, which allow it to generalize when learning context-free languages (Wiles and Elman, 1995; Rodriguez and Wiles, 1998; Rodriguez et al., 1999; Bod´en et al., 1999).”