Learning Explanatory Rules from Noisy Data

By Richard Evans, Edward Grefenstette. 2017

557 citations recorded by the source corpus.

Artificial Neural Networks are powerful function approximators capable of modelling solutions to a wide variety of problems, both supervised and unsupervised. As their size and expressivity increases, so too does the variance of the model, yielding a nearly ubiquitous overfitting problem. Although mitigated by a variety of model regularisation methods, the common cure is to seek large amounts of training data—which is not necessarily easily obtained—that sufficiently approximates the data distribution of the domain we wish to test on. In contrast, logic programming methods such as Inductive Logic Programming offer an extremely data-efficient process by which models can be trained to reason on symbolic domains. However, these methods are unable to deal with the variety of domains neural networks can be applied to: they are not robust to noise in or mislabelling of inputs, and perhaps more importantly, cannot be applied to non-symbolic domains where the data is ambiguous, such as operating on raw pixels. In this paper, we propose a Differentiable Inductive Logic framework (∂ILP), which can not only solve tasks which traditional ILP systems are suited for, but shows a robustness to noise and error in the training data which ILP cannot cope with. Furthermore, as it is trained by backpropagation against a likelihood objective, it can be hybridised by connecting it with neural networks over ambiguous data in order to be applied to domains which ILP cannot address, while providing data efficiency and generalisation beyond what neural networks on their own can achieve.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, background): “Systems that can only acquire rules but not exceptions (e.g. Evans and Grefenstette (Evans & Grefenstette, 2017)]) are an interesting step along the way to building systems that can acquire abstract knowledge, but not sufficient.”
Neural Logic Machines (2019, methodology): “Last but not least, unlike ILP methods that learn a set of rules in an explainable format, the learned rules of NLMs are implicitly encoded as weights of the neural networks.”
Reconciling deep learning with symbolic artificial intelligence: representing objects and relations (2019, background): “Various recent attempts have been made to mimic such processes in neural networks [48–50].”
Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoning (2019, methodology): “Neural-symbolic systems have been applied successfully to several ﬁelds, including data science, ontology learning, training and assessment in simulators, and models of cognitive learning and reasoning [5, 14, 16, 34].”
DeepProbLog: Neural Probabilistic Logic Programming (2018, background): “Finally, Evans and Grefenstette (2018) introduce a differentiable framework for rule induction, that does not focus on the integration of the two approaches like DeepProblog.”
Relational inductive biases, deep learning, and graph networks (2018, background): “We are optimistic about a number of other relevant, and perhaps underappreciated, research directions, including marrying learning-based approaches with programs (Ritchie et al., 2016; Andreas et al., 2016; Gaunt et al., 2016; Evans and Grefenstette, 2018; Evans et al., 2018), developing model-based”
Computing Optimal Decision Sets with SAT (2020, background): “Hence the rising interest in explainable artificial intelligence (XAI) [4, 14, 16, 18, 22, 23, 26, 29, 30, 36–38, 42, 43, 48, 49, 52], including research programs [3, 24] and legislation [17, 21].”
NLProlog: Reasoning with Weak Unification for Question Answering in Natural Language (2019, methodology): “Moreover, such models tend to require large amounts of training data to generalise correctly, and incorporating background knowledge is still an open problem (Rocktäschel et al., 2015; Weissenborn et al., 2017a; Rocktäschel and Riedel, 2017; Evans and Grefenstette, 2017).”
Chain of Explanation: New Prompting Method to Generate Quality Natural Language Explanation for Implicit Hate Speech (2022, background): “Human Evaluation - Informativeness andClarityHuman perceptions have always been considered one of the most important evaluation criteria for Natural Language Generation tasks [9, 12].”
Neural, symbolic and neural-symbolic reasoning on knowledge graphs (2021, background): “We take the example of learning which natural numbers are even from (Evans and Grefenstette, 2018) to explain ILP.”
Abduction-Based Explanations for Machine Learning Models (2018, background): “2017), but also efforts to explain predictions made by such models (Evans and Grefenstette 2018; Li et al. 2018; Ribeiro, Singh, and Guestrin 2018; Ignatiev et al. 2018; Montavon, Samek, and Müller 2018; Ribeiro, Singh, and Guestrin 2016; Lakkaraju, Bach, and Leskovec 2016; Baehrens et al. 2010).”
OptTyper: Probabilistic Type Inference by Optimising Logical and Natural Constraints (2020, unknown): “For deep learning, this logic is also important as it allows to perform back-propagation [Evans and Grefenstette 2018].”
Neurosymbolic AI: the 3rd wave (2020, background): “…to neural-symbolic integration, so-called embedding techniques seek to transform symbolic representations into vector spaces where reasoning can take place through matrix computations over distance functions (Serafini and d’Avila Garcez 2016; Evans and Grefenstette 2018; Yang et al. 2017).”
Neural Learning of One-of-Many Solutions for Combinatorial Problems in Structured Output Spaces (2020, background): “Recent research has also explored their application for solving symbolic reasoning tasks, requiring higher level inferences, such as neural theorem proving [Rocktäschel et al., 2015, Evans and Grefenstette, 2018, Minervini et al., 2020], and playing blocks world [Dong et al., 2019].”
Neural Logic Reinforcement Learning (2019, methodology): “To address this challenge, recently Differentiable Inductive Logic Programming (DILP) has been proposed in which a learning model expressed by logic states can be trained by gradient-based optimization methods (Evans & Grefenstette, 2018; Rockt¨aschel & Riedel, 2017; Cohen et al., 2017).”
DRUM: End-To-End Differentiable Rule Mining On Knowledge Graphs (2019, methodology): “Also dILP [14], unlike our method requires negative examples which is hard to obtain under Open World Assumption (OWA) of modern KGs and dILP is memory-expensive as authors admit, which cannot scale to the size of large KGs, thus we can not compare numerical results here.”
CLN2INV: Learning Loop Invariants with Continuous Logic Networks (2019, methodology): “Unfortunately, existing methods for extracting logical rules from general neural architectures lack sufﬁcient precision (Au-gasta & Kathirvalavakumar, 2012), while inductive logic learning lacks sufﬁcient expressiveness for use in veriﬁcation (Evans & Grefenstette, 2018).”
FHI Oxford Technical Report # 2018-2 Predicting Human Deliberative Judgments with Machine Learning (2018, background): “Second, deliberation can be modeled explicitly by building on recent deep learning approaches [28, 29, 30, 31, 32, 33, 34, 35, 36, 37].”
Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization (2019, background): “Neural models are often considered incapable of such generalization, allegedly because they are limited to the training space (Marcus, 2003; Kaiser and Sutskever, 2015; Reed and De Freitas, 2015; Evans and Grefenstette, 2018).”