Extraction of rules from discrete-time recurrent neural networks

By Christian Walter Peter Omlin, C. Lee Giles. 1996

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, methodology): “…describable by deterministic Finite State Automata (FSAs) (Blair & Pollack, 1997; Casey, 1996; Kalinke & Lehmann, 1998; Manolios & Fanelli, 1994; Omlin & Giles, 1996; Siegelmann, 1992; Vahed & Omlin, 2004; Watrous & Kuhn, 1992; Zeng, Goodman, & Smyth, 1994), both in terms of reliability and…”
Hybrid neural systems: from simple coupling to fully integrated neural networks (1999, background): “The symbolic knowledge is often represented as rules but automata [28, 29, 67, 68, 99] and grammars may also be used.”
Extracting rules from trained neural networks (2000, methodology): “Most algorithms are applied to three-layer feedforward networks, while a few algorithms can be applied only to recurrent neural networks, where DFA’s (deterministic finite-state automata) are extracted [10].”
Discovering the Compositional Structure of Vector Representations with Role Learning Networks (2019, background): “Most closely related are the approaches of [2], [5], and [1], who also propose methods for uncovering symbolic structure in vector representations, and [14] and [20], which also seek to understand neural networks by extracting more interpretable symbolic models that approximate their behavior.”
Recurrent neural networks for time series classification (2003, background): “This means that after training, interrelations between the current input and internal states are processed to produce the output and to represent the relevant past information in the internal states [5,10,15].”
A Machine Learning Method for Extracting Symbolic Knowledge from Recurrent Neural Networks (2004, methodology): “…a study of fixed points of the logistic sigmoid function (Omlin, Thornber, & Giles, 1998; Omlin & Giles, 1996a, 1996c) and algorithms that construct clusters that eventually denote the states in the network (Cleeremans, Servan-Schreiber, & McClelland, 1989; Das & Das, 1992; Omlin & Giles, 1996a).”
Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamples (2017, methodology): “Indeed, several lines of work attempt to extract clear rules for their decisions (Jacobsson, 2005; Omlin & Giles, 1996; Cechin et al., 2003).”
The truth will come to light: directions and challenges in extracting the knowledge embedded within trained artificial neural networks (1998, background): “In addition, both Towell and Shavlik [42], and Omlin and Giles [26] have identified situations in which the extracted knowledge representation (viz. rules in the case of Towell and Shavlik, deterministic finite-state automata [DFA’s] in the case of Omlin and Giles) outperformed the ANN from which th”
Explainable Artificial Intelligence: a Systematic Review (2020, methodology): “[266] and [267] proposed two methods for replacing a DNN with a deterministic finite automaton that can be visualized as a graph where each node represent a cluster of values in the output space and the edges represent the presence of shared patterns in a network’s internal layers between these clus”
An Empirical Evaluation of Rule Extraction from Recurrent Neural Networks (2017, methodology): “…& Kuhn, 1992; Omlin & Giles, 1996c; Casey, 1996; Jacobsson, 2005) has shown that it is possible to extract deterministic ﬁnite automata (DFA) from RNN models trained to perform grammatical inference and that grammatical rules can be stably encoded in second order RNNs (Omlin & Giles, 1996a,b).”
Symbolic Interpretation of Artificial Neural Networks (1999, methodology): “Omlin and Giles [25] have developed a heuristic algorithm to extract grammar rules in the form of Deterministic Finite-state Automata (DFA) from discrete-time neural networks and specifically from second-order networks.”
Learning Finite State Representations of Recurrent Policy Networks (2018, background): “There have been efforts made in the past to understand the internals of Recurrent Networks (Karpa-thy et al., 2015; Arras et al., 2017; Strobelt et al., 2016; Murdoch & Szlam, 2017; Jacobsson, 2005; Omlin & Giles, 1996).”
Noisy Time Series Prediction using Recurrent Neural Networks and Grammatical Inference (2001, background): “The use of a recurrent neural network is significant for two reasons: firstly, the temporal relationship of the series is explicitly modeled via internal states, and secondly, it is possible to extract rules from the trained recurrent networks in the form of deterministic finite state automata [44]2”
Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunities (2021, unknown): “Rule extraction from ML models has been studied for a long time [99, 100, 101].”