Large-Scale Visual Relationship Understanding

By Ji Zhang, Yannis Kalantidis, Marcus Rohrbach, Manohar Paluri, Ahmed M. Elgammal, Mohamed Elhoseiny. 2018

177 citations recorded by the source corpus.

Large scale visual understanding is challenging, as it requires a model to handle the widely-spread and imbalanced distribution of 〈subject, relation, object〉 triples. In real-world scenarios with large numbers of objects and relations, some are seen very commonly while others are barely seen. We develop a new relationship detection model that embeds objects and relations into two vector spaces where both discriminative capability and semantic affinity are preserved. We learn a visual and a semantic module that map features from the two modalities into a shared space, where matched pairs of features have to discriminate against those unmatched, but also maintain close distances to semantically similar ones. Benefiting from that, our model can achieve superior performance even when the visual entity categories scale up to more than 80,000, with extremely skewed class distribution. We demonstrate the efficacy of our model on a large and imbalanced benchmark based of Visual Genome that comprises 53,000+ objects and 29,000+ relations, a scale at which no previous work has been evaluated at. We show superiority of our model over competitive baselines on the original Visual Genome dataset with 80,000+ categories. We also show state-of-the-art performance on the VRD dataset and the scene graph dataset which is a subset of Visual Genome with 200 categories.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

On Mutual Information Maximization for Representation Learning (2019, methodology): “When using different encoders for different views and asymmetric critics as employed by CPC, DeepInfoMax, and CMC one recovers asymmetric variants of (5), see, e.g. (Yu et al., 2017; Zhang et al., 2019).”
Efficient Lifelong Learning with A-GEM (2018, neutral): “Borrowing ideas from literature in few-shot learning (Lampert et al., 2014; Zhang et al., 2018; Elhoseiny et al., 2017; Xian et al., 2018), we learn a joint embedding space between image features and the attribute embeddings.”
Exploring the Challenges towards Lifelong Fact Learning (2018, background): “This can help semantically similar concepts to learn better from one another, as shown in [31,7] in non LLL scenarios.”
DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue (2019, methodology): “The visual relationship encoder (Zhang et al. 2019), which is pre-trained on a visual relationship benchmark, i.e. GQA (Hud-son and Manning 2019), encodes relationships between the subject v i and object v j as a 512 -dimensional relation embedding, denoted as r ij .”
Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identification (2020, methodology): “State-of-the-art UDA methods (Song et al., 2018; Zhang et al., 2019b; Yang et al., 2019) for person re-ID group unannotated images with clustering algorithms and train the network with clustering-generated pseudo labels.”
Multi-Modal Knowledge Graph Construction and Application: A Survey (2022, background): “There are much other work focusing on detecting relations with few samples by transfer learning [86], fewshot learning [87] and contrastive learning [88], still being”
Character Matters: Video Story Understanding with Character-Aware Relations (2020, unknown): “We use the approach by Zhang et al. (2019a) to detect relationships in each frame.”
Semantic Communications: Principles and Challenges (2021, background): “The relationship detection model in [45] learns a module to map features from the vision and semantic modalities into a shared space, where the matched feature pairs should be discriminative against those unmatched ones and maintaining close distances to semantically similar ones.”
ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes (2020, methodology): “(e.g., [40,72,1]) has been explored using datasets such as VRD [40] and more recently on the large Visual Genome dataset [32].”
Visual Relation Grounding in Videos (2020, background): “problem is mostly tackled in the form of detection task and the advancements are mostly made in the image domain [17,20,45].”
Normalization Matters in Zero-Shot Learning (2020, methodology): “2 Understanding “normalize + scale” trick One of the most popular “tricks” among zero-shot learning practitioners is the normalization of vectors z and fc to a unit norm followed by additional scaling afterwards [38, 53].”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, background): “These applications include visual-textual transformers [139], [140], [141], image text retrieval [3], [6], [142], visual question answering [143], [144], image understanding and reasoning [145], [146], [147], 3D scene graphs [7], [148], and the detection and recognition of human-object and human-hum”
Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation (2020, background): “(10) Our main finding is that, while these losses can give similar or better results in some cases, the parameters α, β and λ do not generally transfer across datasets and must be tuned every time, which can be problematic at larger scale [44].”
Efficient Lifelong Learning with A-GEM (2018, unknown): “Borrowing ideas from literature in few-shot learning (Lampert et al., 2014; Zhang et al., 2018; Elhoseiny et al., 2017; Xian et al., 2018), we learn a joint embedding space between image features and the attribute embeddings.”