Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation

By Ruichi Yu, Ang Li, Vlad I. Morariu, Larry S. Davis. IEEE International Conference on Computer Vision, 2017

326 citations recorded by the source corpus.

Understanding the visual relationship between two objects involves identifying the subject, the object, and a predicate relating them. We leverage the strong correlations between the predicate and the hsubj; obji pair (both semantically and spatially) to predict predicates conditioned on the subjects and the objects. Modeling the three entities jointly more accurately reflects their relationships compared to modeling them independently, but it complicates learning since the semantic space of visual relationships is huge and training data is limited, especially for longtail relationships that have few instances. To overcome this, we use knowledge of linguistic statistics to regularize visual model learning. We obtain linguistic knowledge by mining from both training annotations (internal knowledge) and publicly available text, e.g., Wikipedia (external knowledge), computing the conditional probability distribution of a predicate given a (subj, obj) pair. As we train the visual model, we distill this knowledge into the deep model to achieve better generalization. Our experimental results on the Visual Relationship Detection (VRD) and Visual Genome datasets suggest that with this linguistic knowledge distillation, our model outperforms the stateof- the-art methods significantly, especially when predicting unseen relationships (e.g., recall improved from 8.45% to 19.17% on VRD zero-shot testing set).

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/fdc0754852b9c8366341972f1b5b4320b48d64a9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Semantic Image Interpretation - Integration of Numerical Data and Logical Knowledge for Cognitive Vision (2018, unknown): “Now it is second to a Computer Vision top conference paper [109].”
Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation (2018, methodology): “Moreover, spatial information has been shown to be valuable in predicate recognition [6, 8, 9].”
Character Matters: Video Story Understanding with Character-Aware Relations (2020, unknown): “Most recent approaches achieve this goal by learning classiﬁers that predict relations based on different types of features of the object pairs (Xu et al. 2017; Yu et al. 2017; Zellers et al. 2018; Zhang et al. 2019a; 2019b).”
Revisiting Knowledge Distillation via Label Smoothing Regularization (2020, methodology): “Knowledge Distillation Since [7] proposed knowledge distillation based on prior work [2], KD has been widely adopted or modified [14, 19, 20, 4, 1, 11, 17].”
Neural Motifs: Scene Graph Parsing with Global Context (2017, methodology): “0 42.3 48.2 MSDN-FREQ 13.5 15.7 25.8 27.8 56.0 61.0 SCR[23] 10.67 13.81 16.58 21.54 split DR-NET[6] 20.79 23.76 23.95 27.57 88.26 91.26 VRL[27] 12.57 13.34 14.36 16.09 VTE[57] 5.52 6.04 9.46 10.45 LKD[54] 92.31 95.68 Table 3. Results with and without scene graph constraints. Horizontal lines indicat”
Learning Prototypes for Visual Relationship Detection (2018, methodology): “KD [10] is a network trained with a loss function augmented with knowledge from the dataset statistics, and KD - no stat is the same network trained without this knowledge.”
Beyond Short-Term Snippet: Video Relation Detection With Spatio-Temporal Global Context (2020, background): “The most common approach to detect visual relations is to ﬁrst ﬁnd all objects in the image and predict relationship between each pair of them, as in [24, 46, 48, 51, 9].”
Neural Compatibility Modeling with Attentive Knowledge Distillation (2018, background): “Later, Yu et al. [33] proposed to utilize the knowledge of linguistic statistics to regularize the learning process in the context of visual relationship detection.”
Towards Understanding Knowledge Distillation (2019, background): “…et al., 2016), compression (Howard et al., 2017; Polino et al., 2018), integration with ﬁrst-order logic (Hu et al., 2016) or other prior knowledge (Yu et al., 2017), learning from noisy labels (Li et al., 2017), defending against adversarial attacks (Papernot et al., 2016), training…”
NISP: Pruning Networks Using Neuron Importance Score Propagation (2017, background): “Despite their impressive predictive power on a wide range of tasks [33, 40, 41, 13, 15, 48, 46, 24, 44, 42, 12, 45], the redundancy in the parameterization of deep learning models has been studied and demonstrated [6].”
LinkNet: Relational Embedding for Scene Graph (2018, unknown): “This includes detection of human-object interactions [7, 3], localization of proposals from natural language expressions [12], or the more general tasks of visual relationship detection [17, 25, 38, 5, 19, 37, 34, 41] and scene graph generation [33, 18, 35, 22].”
Learning Visual Commonsense for Robust Scene Graph Generation (2020, methodology): “Incorporating commonsense knowledge has been explored in various computer vision tasks such as object recognition [3,28,14], object detection [13], semantic segmentation [19], action recognition [9], visual relation detection [31], scene graph generation [2,34,7], and visual question answering [18,2”
Knowledge Distillation via Instance Relationship Graph (2019, background): “Therefore, more and more efforts have been putting into Knowledge distillation [9, 26, 16, 25] is one of the most popular solutions for model compression.”
Scene Graph Generation With External Knowledge and Image Reconstruction (2019, background): “Unlike [45], which leverages linguistic knowledge to regularize the network, our knowledge-based module improves the feature refining procedure by reasoning over a basket of commonsense knowledge retrieved from ConceptNet.”
Know More Say Less: Image Captioning Based on Scene Graphs (2019, background): “[52] obtain linguistic knowledge by mining from both training annotations and publicly available text to regularize visual model learning.”
Self-Distillation as Instance-Specific Label Smoothing (2020, methodology): “First introduced as a simple method to compress high-capacity neural networks into a low-capacity counterpart for computational efficiency, knowledge distillation [15] has since gained much popularity across various application domains ranging from computer vision to natural language processing [19,”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, background): “For this reason, LKD (Linguistic Knowledge Distillation) [19] uses not only the annotations inside the training set but also text publicly available on the Internet (Wikipedia) to collect external language knowledge.”
Improved Knowledge Distillation via Teacher Assistant (2019, background): “To solve this issue, Yim et al.; Yu et al. (2017; 2017) used a shared representation of layers, however, it’s not straight-forward to choose the appropriate layer to be matched.”