Differentiable Reasoning on Large Knowledge Bases and Natural Language

By Pasquale Minervini, Matko Bovsnjak, Tim Rocktaschel, Sebastian Riedel, Edward Grefenstette. 2019

98 citations recorded by the source corpus.

Reasoning with knowledge expressed in natural language and Knowledge Bases (KBs) is a major challenge for Artificial Intelligence, with applications in machine reading, dialogue, and question answering. General neural architectures that jointly learn representations and transformations of text are very data-inefficient, and it is hard to analyse their reasoning process. These issues are addressed by end-to-end differentiable reasoning systems such as Neural Theorem Provers (NTPs), although they can only be used with small-scale symbolic KBs. In this paper we first propose Greedy NTPs (GNTPs), an extension to NTPs addressing their complexity and scalability limitations, thus making them applicable to real-world datasets. This result is achieved by dynamically constructing the computation graph of NTPs and including only the most promising proof paths during inference, thus obtaining orders of magnitude more efficient models. Then, we propose a novel approach for jointly reasoning over KBs and textual mentions, by embedding logic facts and natural language sentences in a shared embedding space. We show that GNTPs perform on par with NTPs at a fraction of their cost while achieving competitive link prediction results on large datasets, providing explanations for predictions, and inducing interpretable models. Source code, datasets, and supplementary material are available online at this https URL.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, background): “Papers like Minervini et al (Minervini et al., 2019) and Yang et al (Yang, Yang, & Cohen, 2017) have made real progress towards building end-to-end differentiable hybrid neurosymbolic systems that work at scale.”
RuleBERT: Teaching Soft Rules to Pre-Trained Language Models (2021, methodology): “Unlike previous work (Hamilton et al., 2018; Yang et al., 2017; Minervini et al., 2020), we do not design a new, ad-hoc module for neural reasoning, but we rely solely on the transformer’s capability to emulate algorithms (Wang et al., 2019b; Lample and Charton, 2020).”
Transformers as Soft Reasoners over Language (2020, methodology): “Several researchers have developed methods for Neural Theorem Proving (NTP), combining symbolic and neural methods to reason step-wise over language-derived structures, e.g., [Weber et al. , 2019; Minervini et al. , 2019; Minervini et al. , 2018].”
Neurosymbolic AI: the 3rd wave (2020, background): “…and d’Avila Garcez 2016) takes another step towards integration by using a differentiable many-valued logic in the loss function of a neural network; (Minervini et al. 2020; Rocktäschel and Riedel 2017) proposes to perform differentiable unification and theorem proving inside the neural network.”
Neural Learning of One-of-Many Solutions for Combinatorial Problems in Structured Output Spaces (2020, background): “ch has also explored their application for solving symbolic reasoning tasks, requiring higher level inferences, such as neural theorem proving [Rocktäschel et al., 2015, Evans and Grefenstette, 2018, Minervini et al., 2020], and playing blocks world [Dong et al., 2019]. The advantage of neural model”
Exploring End-to-End Differentiable Natural Logic Modeling (2020, methodology): “For example, the work in (Rocktäschel and Riedel, 2017; Weber et al., 2019; Minervini et al., 2020) proposes a differentiable backward-chaining algorithm, and Dong et al.”
Neural Databases (2020, background): “Most closely to our work, Minervini et al. [28] has showed how differentiable prolog interpreters can be used to support reasoning with facts in natural language.”
Relational Memory-Augmented Language Models (2022, background): “, 2021), machine reading (Yang and Mitchell, 2019), and reasoning (Minervini et al., 2020).”
Knowledge Graphs: Opportunities and Challenges (2023, background): “Tremendous research efforts have focused on reasoning to enrich the knowledge graphs, which aims to infer new facts based on existing data (Minervini et al. 2020).”