The More You Know: Using Knowledge Graphs for Image Classification

By Kenneth Marino, Ruslan Salakhutdinov, Abhinav Kumar Gupta. 2016

379 citations recorded by the source corpus.

One characteristic that sets humans apart from modern learning-based computer vision algorithms is the ability to acquire knowledge about the world and use that knowledge to reason about the visual world. Humans can learn about the characteristics of objects and the relationships that occur between them to learn a large variety of visual concepts, often with few examples. This paper investigates the use of structured prior knowledge in the form of knowledge graphs and shows that using this knowledge improves performance on image classification. We build on recent work on end-to-end learning on graphs, introducing the Graph Search Neural Network as a way of efficiently incorporating large knowledge graphs into a vision classification pipeline. We show in a number of experiments that our method outperforms standard neural network baselines for multi-label classification.

Subfields: Computer Vision and Pattern Recognition, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Graph Learning

Citations in the corpus

Neural Message Passing for Quantum Chemistry (2017, methodology): “Recent work has adapted the GG-NN architecture to larger graphs by passing messages on only subsets of the graph at each time step (Marino et al., 2016).”
Knowledge Aware Semantic Concept Expansion for Image-Text Matching (2019, methodology): “Knowledge can be used for image tasks like image classification [Marino et al., 2017], zero shot classification [Lee et al., 2018a] and for text tasks [Zhou et al., 2018].”
Deep Learning on Graphs: A Survey (2018, neutral): “Besides standard graph inference tasks such as node or graph classification, graph-based deep learning methods have also been applied to a wide range of disciplines, such as modeling social influence [103], recommendation [51], [78], [96], chemistry [37], [41], [50], [98], [99], physics [104], [105]”
Visual Semantic Navigation using Scene Priors (2018, methodology): “A knowledge graph is used by Marino et al. (2017) for image classiﬁcation, by Li et al. (2017) for situation recognition and by Wang et al. (2018) for zero-shot recognition.”
DOCK: Detecting Objects by Transferring Common-Sense Knowledge (2018, methodology): “Such external knowledge bases have been used in several vision tasks including image classification [25,7], VQA [47,41], visual relationship detection [24,30],”
Learning Human-Object Interactions by Graph Parsing Neural Networks (2018, background): “Some other work integrated network architectures with graphical models [12,20] and gained promising results on applications such as scene understanding [24,30, 46], object detection and parsing [27,50], and VQA [41].”
Learning from the Past: Meta-Continual Learning with Knowledge Embedding for Jointly Sketch, Cartoon, and Caricature Face Recognition (2020, positive): “GGNN is trained with ADAM following [28].”
Symbolic Graph Reasoning Meets Convolutions (2018, methodology): “[32] used structure prior knowledge to enhance predictions of multi-label classification while our SGR proposes a general neural layer that can be injected into any convolution layers and allows the neural network to leverage semantic constraints derived from various human knowledge.”
Videos as Space-Time Region Graphs (2018, methodology): “Instead of using mean-field inference, variant simpler feedforward graph based neural network have been proposed recently [67,68,19,69,70,71].”
Multi-label Zero-Shot Learning with Structured Knowledge Graphs (2017, methodology): “In GSNN, the structured knowledge graph is defined with around 30 relation types.”
Deep Reasoning with Knowledge Graph for Social Relationship Understanding (2018, methodology): “During training, all components of the model are trained with SGD except that the GGNN is trained with ADAM following [ Marino et al. , 2016 ] .”
Towards a Universal Continuous Knowledge Base (2020, background): “, 2008) have become a typical example of knowledge base in model AI systems recently, showing their effectiveness in applications such as image classification (Marino et al., 2017), natural language processing (Zhang et al.”
Graph Partition Neural Networks for Semi-Supervised Classification (2018, background): “They can be roughly categorized into generalizations of recurrent neural networks (RNNs) [13, 14, 21, 22, 25, 28, 29, 34, 37] and generalizations of convolutional neural networks (CNNs)”
Bayesian Relational Memory for Semantic Visual Navigation (2019, background): “Similar ideas of using semantic knowledge to enhance reasoning have been applied to image classification [34, 56], segmentation [52, 64], situation recognition [33], visual question answering [57, 9, 53, 28, 26], image retrieval [29] and relation detection [62].”
Iterative Visual Reasoning Beyond Convolutions (2018, background): “Our framework, while being more general, is along the line of research that applies visual knowledge base to end tasks, such as affordances [56], image classification [32], or question answering [49].”
Structure Inference Net: Object Detection Using Scene-Level Context and Instance-Level Relationships (2018, background): “Several interesting works [28, 34, 23, 39, 21, 9, 2, 22, 42] have been proposed to combine deep networks with graphical models for structured prediction tasks that are solved by structure inference techniques.”
Knowledge-Embedded Routing Network for Scene Graph Generation (2019, methodology): “Note that SMN is the previous best-performing method. and thus facilitates various vision tasks ranging from fundamental recognition and detection [20, 8] to high-level tasks [34, 32].”
Informed Machine Learning – A Taxonomy and Survey of Integrating Prior Knowledge into Learning Systems (2019, methodology): “This allows for multi-label classification in natural images where inference about a particular object is facilitated by using relations to other objects in an image [15] (see Insert 4).”
Inference in Probabilistic Graphical Models by Graph Neural Networks (2018, background): “For this reason, GNNs are powerful architectures for capturing statistical dependencies between variables of interest (Bruna et al., 2014; Duvenaud et al., 2015; Li et al., 2016; Marino et al., 2016; Li et al., 2017; Qi et al., 2017; Kipf & Welling, 2017).”