Visual Translation Embedding Network for Visual Relation Detection

By Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, Tat-seng Chua. 2017

603 citations recorded by the source corpus.

Visual relations, such as person ride bike and bike next to car, offer a comprehensive scene understanding of an image, and have already shown their great utility in connecting computer vision and natural language. However, due to the challenging combinatorial complexity of modeling subject-predicate-object relation triplets, very little work has been done to localize and predict visual relations. Inspired by the recent advances in relational representation learning of knowledge bases and convolutional object detection networks, we propose a Visual Translation Embedding network (VTransE) for visual relation detection. VTransE places objects in a low-dimensional relation space where a relation can be modeled as a simple vector translation, i.e., subject + predicate ≈ object. We propose a novel feature extraction layer that enables object-relation knowledge transfer in a fully-convolutional fashion that supports training and inference in a single forward/backward pass. To the best of our knowledge, VTransE is the first end-toend relation detection network. We demonstrate the effectiveness of VTransE over other state-of-the-art methods on two large-scale datasets: Visual Relationship and Visual Genome. Note that even though VTransE is a purely visual model, it is still competitive to the Lu’s multi-modal model with language priors [27].

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

SCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning (2016, methodology): “We are interested in visual attention models used in the encoder-decoder framework for neural image/video captioning (NIC) and visual question answering (VQA), which fall into the recent trend of connecting computer vision and natural language [14, 41, 24, 23, 42, 12].”
Auto-Encoding Scene Graphs for Image Captioning (2018, background): “By observing the potential of exploiting scene graphs in vision tasks, a variety of approaches are proposed to improve the scene graph generation from images [57, 56, 52, 51, 49, 42, 6].”
Neural Motifs: Scene Graph Parsing with Global Context (2017, background): “Others have explored creating richer models between image regions, introducing new convolutional features and new objectives [28, 50].”
A Domain Based Approach to Social Relation Recognition (2017, background): “Social relation is different with the visual relation of “subject-predicate-object” [33, 34], but it is derived from human social behaviors.”
Unbiased Scene Graph Generation From Biased Training (2020, positive): “VTransE [71] uses fully connected layers and element-wise subtraction for feature merging.”
Neural Compatibility Modeling with Attentive Knowledge Distillation (2018, methodology): “In our setting, we focused on the average position of the positive bottom in the ranking list and thus adopted the mean reciprocal rank (MRR) metric [22, 41, 44].”
Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation (2018, background): “As an alternative paradigm, considering relationship predicates and object categories separately becomes more popular in recent works [7,9,51,52].”
Learning to Detect Human-Object Interactions (2017, background): “As a result, follow-up works [2, 21, 20, 37, 38] Figure 2: Generating human-object proposals from human and object detections.”
Learning to Collocate Neural Modules for Image Captioning (2019, background): “(1) as the input because these kinds of features are widely applied as the input for successful relationship detection [61, 59].”
Learning to Assemble Neural Module Tree Networks for Visual Grounding (2018, background): “Thanks to the development of deep visual features [9] and language models [27], we can extend the grounding systems from fixed-size inventory of words to open-vocabulary [12] or even descriptive and relational phrases [41, 31].”
Visual Semantic Navigation using Scene Priors (2018, methodology): “he current observation and the prior knowledge as it moves in the environment. Object relationships have been used for tasks such as image retrieval (Johnson et al., 2015), visual relation detection (Zhang et al., 2017), referring expressions (Nagaraja et al., 2016; Hu et al., 2017), and visual ques”
Semantic Communications: Principles and Challenges (2021, background): “Particularly, VTransE places objects in a low-dimension relation space where the relationship can be modeled as a simple translation vector (subject + predicate = object) [44].”
Large-Scale Visual Relationship Understanding (2018, methodology): “Almost all of them are based on a small vocabulary, e.g., 100 object and 70 relation categories from the VRD dataset[21], or a subset of VG with the most frequent object and relation categories (e.g., 200 object and 100 relation categories from [37]).”
Relation-Aware Graph Attention Network for Visual Question Answering (2019, methodology): “A few neural network architectures have been designed for the visual relationship prediction task [32, 8, 61].”
3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans (2020, unknown): “answering [27, 116], and action detection [59, 64, 111].”
LinkNet: Relational Embedding for Scene Graph (2018, unknown): “This includes detection of human-object interactions [7, 3], localization of proposals from natural language expressions [12], or the more general tasks of visual relationship detection [17, 25, 38, 5, 19, 37, 34, 41] and scene graph generation [33, 18, 35, 22].”
Attention Residual Learning for Skin Lesion Classification (2019, unknown): “It has made great progress in the cross ﬁelds of computer vision and natural language processing, such as image/video caption and visual question answering [14], [15], [16], [17], [18].”
Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, background): “Furthermore, pairing different objects in a scene [13, 28, 60, 78] is also common.”