Logic Tensor Networks for Semantic Image Interpretation

By Ivan Donadello, Luciano Serafini, A.S. d'Avila Garcez. 2017

246 citations recorded by the source corpus.

Semantic Image Interpretation (SII) is the task of extracting structured semantic descriptions from images. It is widely agreed that the combined use of visual data and background knowledge is of great importance for SII. Recently, Statistical Relational Learning (SRL) approaches have been developed for reasoning under uncertainty and learning in the presence of data and rich knowledge. Logic Tensor Networks (LTNs) are an SRL framework which integrates neural networks with first-order fuzzy logic to allow (i) efficient learning from noisy data in the presence of logical constraints, and (ii) reasoning with logical formulas describing general properties of the data. In this paper, we develop and apply LTNs to two of the main tasks of SII, namely, the classification of an image's bounding boxes and the detection of the relevant part-of relations between objects. To the best of our knowledge, this is the first successful application of SRL to such SII tasks. The proposed approach is evaluated on a standard image processing benchmark. Experiments show that the use of background knowledge in the form of logical constraints can improve the performance of purely data-driven approaches, including the state-of-the-art Fast Region-based Convolutional Neural Networks (Fast R-CNN). Moreover, we show that the use of logical background knowledge adds robustness to the learning system when errors are present in the labels of the training data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Knowledge Graphs & Representation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Knowledge Graphs & Representation

Citations in the corpus

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, positive): “Therefore, neural-symbolic interpretability could provide convincing explanations while keeping or improving generic performance [297] .”
Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoning (2019, methodology): “In [12], a Fast-RCNN [18] is used for bounding-box detection of parts of objects and on top of that, a Logic Tensor Network is used to reason about relations between parts of objects and types of such objects.”
Integrating Learning and Reasoning with Deep Logic Models (2019, background): “Unlike in Semantic-based Regularization [8] or Logic Tensor Networks [9], the sensorial and reasoning layers can be jointly trained, so that the high-level weights imposing the output structure are jointly learned together with the neural network weights, processing the low-level input.”
Exploring End-to-End Differentiable Natural Logic Modeling (2020, methodology): “One uses symbolic constraints as regularizers to equip neural models with the corresponding inductive bias (Demeester et al., 2016; Diligenti et al., 2017; Donadello et al., 2017; Xu et al., 2018; Li and Srikumar, 2019).”
A Semantic Loss Function for Deep Learning with Symbolic Knowledge (2017, background): “, 2017) or logical constraints (Rocktäschel et al., 2015; Hu et al., 2016; Demeester et al., 2016; Stewart & Ermon, 2017; Minervini et al., 2017; Diligenti et al., 2017; Donadello et al., 2017) on the output of a neural network.”
Logic Tensor Networks (2020, methodology): “Secondly, LTN should be compared in large-scale practical use cases with other recent efforts to add structure toneural networks suchas theNeuro-SymbolicConcept Learner [44] and high-level capsules which were used recently to learn the part-of relation [38], similarly to LTN in [18].”
Towards Explainable Neural-Symbolic Visual Reasoning (2019, background): “It has been proven that using a background knowledge within a DNN can bring robustness to the learning system [Donadello et al., 2017; Donadello, 2018; d’Avila Garcez et al., 2019].”
A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and concepts (2021, methodology): “Logic Tensor Nets [38] any b X p/r sym rule”
Neurosymbolic AI: the 3rd wave (2020, unknown): “” Neuralsymbolic computing seeks to offer such a principled way of studying AI by establishing provable correspondences between neural models and logical representations [4, 23, 16, 20, 12].”
Semantic Image Interpretation - Integration of Numerical Data and Logical Knowledge for Cognitive Vision (2018, unknown): “Therefore, it is important to test if such a background knowledge improves the performance of a classical Machine Learning setting, see our works [92, 28].”
Neuro-Symbolic = Neural + Logical + Probabilistic (2019, methodology): “For instance, [Diligenti et al., 2017] and [Donadello et al., 2017] use first-order logic to specify constraints on the output of the neural network, while [Demeester et al., 2016] and [Minervini et al., 2017] use logical IF-THEN rules, derived from expert knowledge, to enforce the embeddings to be…”
Logical Rule Induction and Theory Learning Using Neural Theorem Proving (2018, methodology): “A follow-up paper applies LTN’s to semantic image segmentation (Donadello, Seraﬁni, and D’Avila Garcez 2017).”