Survey and critique of techniques for extracting rules from trained artificial neural networks

By Robert Andrews, Joachim Diederich, Alan B. Tickle. 1995

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Explaining Explanations: An Overview of Interpretability of Machine Learning (2018, neutral): “Andrews et al [26] outlines existing rule extraction techniques, and provides a useful taxonomy of ﬁve dimensions of rule-extraction methods including their expressive power, translucency and the quality of rules.”
Stealing Machine Learning Models via Prediction APIs (2016, methodology): “In a non-malicious setting, improper model extraction techniques have been applied for interpreting [2, 19, 52] and compressing [16, 27] complex neural networks.”
Deep learning in neural networks: An overview (2014, background): “LSTM also worked better on tasks involving context free languages (CFLs) that cannot be represented by HMMs or similar FSAs discussed in the RNN literature (Sun et al., 1993b; Wiles and Elman, 1995; Andrews et al., 1995; Steijvers and Grunwald, 1996; Tonkes and Wiles, 1997; Rodriguez et al., 1999; R”
Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoning (2019, background): “Most of them are discussed in the surveys [1, 24, 55].”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, methodology): “The survey by Ras [74] had taken on the classification of rule extraction strategies proposed earlier in [117] and [118] and proposed three modes to extract rules: (a) pedagogical rule extraction, (b) decompositional rule extraction and (c) eclectic Rule-Extraction.”
Local Rule-Based Explanations of Black Box Decision Systems (2018, background): “Classification rules have been widely used to explain neural networks [2, 3, 21] but also to understand the global behavior of SVMs [15, 27].”
Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges (2018, methodology): “This property is extremely important in safety-critical applications (Andrews et al., 1995) where ambiguity is to be avoided.”
Rule Extraction Algorithm for Deep Neural Networks: A Review (2016, background): “However, safety-critical systems require reliability validation, otherwise it leads to danger [12].”
Designing Interpretable Approximations to Deep Reinforcement Learning with Soft Decision Trees (2020, background): “Evaluation metrics The collection of controller evaluation metrics presented here is most closely related to (Andrews, Diederich, and Tickle 1995), which proposes four metrics to measure the quality of rules extracted from a neural network (Dancey, Bandar, and McLean 2007): accuracy, fidelity,…”
Comprehensible credit scoring models using rule extraction from support vector machines (2007, neutral): “Although rule extraction for neural networks has been extensively researched (a.o. [1, 3]), very little literature is available on SVM rule extractions.”
Meaningful Explanations of Black Box AI Decision Systems (2019, neutral): “…Sivakumar, and Bhattacharya 1999) and tree ensembles (Tan, Hooker, and Wells 2016), while decision rules have been widely used to explain neural networks (Augasta and Kathirvalavakumar 2012; Andrews, Diederich, and Tickle 1995) and SVM (Support Vector Machines) (Fung, Sandilya, and Rao 2005).”
Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review (2020, unknown): “Various papers have developed techniques to explain neural networks and tree ensembles using single decision tree [65, 83, 184] and rule sets [14, 76].”
A review of analytical techniques for gait data. Part 2: neural network and wavelet methods. (2001, methodology): “Alternatives for interpreting weights and biases include neural rule extraction [38,39], Hinton diagrams [40,41] and bond diagrams [42].”
A model explanation system (2016, background): “We do not aim to summarize how the model “works in general” (see [4, 5, 6]), but only seek explanations of individual cases.”
Adversarial Neural Network Inversion via Auxiliary Knowledge Alignment (2019, background): “On the other hand, an important drawback of neural networks is their lack of explanation capability [42].”
Dimensions of Neural-symbolic Integration - A Structured Survey (2005, background): “Some of the other eﬀorts are reported in [Towell and Shavlik, 1993, Andrews et al. , 1995, Bologna, 2000, Garcez et al. , 2001, Lehmann et al. , 2005] It shall be noted that only a few systems have been proposed to date which include representation, learning, and extraction capabilities in a…”