The Dynamics of Discrete-Time Computation, with Application to Recurrent Neural Networks and Finite State Machine Extraction

By Michael Casey. 1996

Subfields: Model Architectures, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Statistical Learning Theory

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, background): “A long tradition in work on neural networks is to evaluate and analyze their ability to learn different formal languages (Das et al., 1992; Casey, 1996; Gers and Schmidhuber, 2001; Bodén and Wiles, 2002; Chalup and Blair, 2003).”
Deep learning in neural networks: An overview (2014, methodology): “LSTM clearly outperformed previous RNNs on tasks that require learning the rules of regular languages describable by deterministic Finite State Automata (FSAs) (Watrous and Kuhn, 1992; Casey, 1996; Siegelmann, 1992; Blair and Pollack, 1997; Kalinke and Lehmann, 1998; Zeng et al., 1994; Manolios and ”
Context-free and context-sensitive dynamics in recurrent neural networks (2000, methodology): “Second, the computational power of RNNs is, in theory, equivalent to that of Turing Machines (Siegelman 1999), implying that RNNs are capable of processing all recursively enumerable languages.”
Learning a context-free task with a recurrent neural network: An analysis of stability (1999, background): “As a string is processed, the hidden unit activations move between these regions in the same manner as the DFA makes transitions between states (Casey, 1996).”
Incremental training of first order recurrent neural networks to predict a context-sensitive language (2003, background): “But if we demand robustness to bounded noise, recurrent neural networks cannot recognize anything beyond regular languages (Casey, 1996; Maass & Orponen, 1997, 1998) and in the case of general Gaussian and other common noise distributions their capabilities are even more restricted (Maass & Sontag,…”
Connectionist sentence processing in perspective (1999, unknown): “(See Cleeremans , 1993; Cleeremans, Servan–Schreiber, & McClelland 1995; Casey, 1996.)”
Extended Dissipative State Estimation for Markov Jump Neural Networks With Unreliable Links (2017, background): “Thereinto, one of these issues is the information latching problem of recurrent NNs (RNNs) [5], [6].”
Rule Extraction from Recurrent Neural Networks: ATaxonomy and Review (2005, methodology): “…(Crutchﬁeld & Young 1990, Servan-Schreiber, Cleeremans & McClelland 1991, Crutchﬁeld 1994, Kolen 1994, Horne & Hush 1994, Siegelmann & Sontag 1995, Casey 1996, Tiˇno, Horne, Giles & Collingwood 1998, Jagota, Plate, Shastri & Sun 1999, Omlin & Giles 2000, Sima & Orponen 2003, Hammer & Tiˇno…”
Simple Recurrent Networks Learn Context-Free and Context-Sensitive Languages by Counting (2001, background): “Although a finite-dimensional RNN cannot process CFLs robustly (Casey, 1996; Maass & Orponen, 1997), this work shows that an SRN can develop the kinds of trajectories that represent a language processing solution.”
Fractal encoding of context‐free grammars in connectionist networks (2000, result): “…the network from responsibility for the challenging part of the task — keeping track of an unbounded memory — thus making its accomplishment fairly similar to another well-studied case, the learning of ﬁnite-state languages (e.g. Servan-Schreiber et al ., 1991; Zheng et al. , 1994; Casey, 1996).”
Exponential stability of delayed recurrent neural networks with Markovian jumping parameters (2006, methodology): “A widely used approach to dealing with the information latching problem is to extract finite state representations (also called clusters, patterns, or modes) from trained networks [2], [5], [6], [7].”
Stability and Synchronization of Discrete-Time Markovian Jumping Neural Networks With Mixed Mode-Dependent Time Delays (2009, background): “This is referred to as the information latching problem [2] that can be handled by extracting finite-state representations (also called clusters, patterns, or modes) from trained networks [3], [7], [8], [11].”
Asynchronous translations with recurrent neural nets (1997, methodology): “A study of (a) the extraction of extended Mealy machines from the trained DTRNN and (b) the conditions under which the augmented DTRNN presented here may “robustly model” [3] an extended Mealy machine is currently under way.”
Exponential synchronization of complex networks with Markovian jump and mixed delays (2008, background): “For example, in [4, 30, 36], it has been revealed that a neural network sometimes has finite modes that switch from one to another at different times, and such a switching (or jumping) can be governed by a Markovian chain.”
Architectural Bias in Recurrent Neural Networks: Fractal Analysis (2002, methodology): “Major research eﬁort has been devoted to the study of learning and implementation of regular grammars and ﬂnite-state transducers using RNNs (Casey, 1996; Cleeremans, Servan-Schreiber & McClelland, 1989; Elman, 1990; For-cada & Carrasco, 1995; Frasconi et al., 1996; Giles et al., 1992; Manolios &…”
A Machine Learning Method for Extracting Symbolic Knowledge from Recurrent Neural Networks (2004, unknown): “The theoretical foundation for DFA extraction based on partitioning of the state-space has been established in Casey (1996).”