Weakly-Supervised Learning of Visual Relations

By Julia Peyre, Ivan Laptev, Cordelia Schmid, Josef Sivic. 2017

200 citations recorded by the source corpus.

This paper introduces a novel approach for modeling visual relations between pairs of objects. We call relation a triplet of the form (subject; predicate; object) where the predicate is typically a preposition (eg. ’under’, ’in front of’) or a verb (’hold’, ’ride’) that links a pair of objects (subject; object). Learning such relations is challenging as the objects have different spatial configurations and appearances depending on the relation in which they occur. Another major challenge comes from the difficulty to get annotations, especially at box-level, for all possible triplets, which makes both learning and evaluation difficult. The contributions of this paper are threefold. First, we design strong yet flexible visual features that encode the appearance and spatial configuration for pairs of objects. Second, we propose a weakly-supervised discriminative clustering model to learn relations from image-level labels only. Third we introduce a new challenging dataset of unusual relations (UnRel) together with an exhaustive annotation, that enables accurate evaluation of visual relation retrieval. We show experimentally that our model results in state-of-theart results on the visual relationship dataset [32] significantly improving performance on previously unseen relations (zero-shot learning), and confirm this observation on our newly introduced UnRel dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Big Transfer (BiT): General Visual Representation Learning (2019, background): “It has been shown that CNNs can lack robustness when classifying objects out-of-context [4,42,47].”
Affine Invariant Analysis of Frank-Wolfe on Strongly Convex Sets (2020, background): “They form a practical family of algorithms [Jaggi 2013; Bojanowski et al. 2014; Alayrac et al. 2016; Seguin et al. 2016; Peyre et al. 2017; Miech x k +1 = (1 − γ t ) x k + γ k v k ⊲ Convex update 5: end for et al. 2018; Lacoste-Julien et al. 2015b; Courty et al. 2016; Paty et al. 2019; Luise et al.…”
The Open Images Dataset V4 (2018, background): “…a fully supervised scenario (Lu et al., 2016; Liang et al., 2017; Dai et al., 2017; Zhang et al., 2017a; Li et al., 2017), in a weakly supervised setting (Peyre et al., 2017; Zhang et al., 2017b), or focusing on human-object relations (Gupta et al., 2009; Yao and Fei-Fei, 2010; Prest et al., 2012).”
Not Using the Car to See the Sidewalk — Quantifying and Controlling the Effects of Context in Classification and Segmentation (2018, methodology): “Additionally we also test our COCO trained models on the UnRel dataset [17] which contains natural images with objects occurring in unusual contexts and relationships.”
Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition (2019, background): “Learning from weak supervision: Given the known challenges in gathering exhaustive annotations for various image and video tasks, leveraging object labels and other meta information to supply weak supervision [7, 15, 19, 22, 30, 31, 39, 40, 50, 52, 53, 54, 55, 56, 59] is a widely-adopted approach.”
Learning Prototypes for Visual Relationship Detection (2018, methodology): “This family of training methods was also recently used for visual relationship detection [13], [16] weakening the underlying assumption that ground truth annotations are complete.”
Projection-Free Optimization on Uniformly Convex Sets (2020, background): “…for instance in large-scale structured prediction (Bojanowski et al., 2014, 2015; Alayrac et al., 2016; Seguin et al., 2016; Miech et al., 2017; Peyre et al., 2017; Miech et al., 2018), quadrature rules in RKHS (Bach et al., 2012; Lacoste-Julien et al., 2015; Futami et al., 2019), optimal…”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, background): “one-by-one detection of the visual relationship of the paired regions [23], [29], [105], [106], [107], [108] is also poorly suited to describing the structure of the entire scene.”
iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection (2018, background): “A number of recent work addresses the problem of detecting visual relationship [1, 7, 19, 22, 23, 32, 43, 44] and generating scene graph [25, 39, 42].”
Learning to Generate Scene Graph from Natural Language Supervision (2021, background): “[35] proposed to assign image-level labels to pairs of detected objects via discriminative clustering.”
Learning De-biased Representations with Biased Representations (2019, methodology): “However, collecting unusual inputs can be expensive (Peyre et al., 2017), and building a generative model with pre-deﬁned bias types (Geirhos et al., 2019) may suffer from bias mis-speciﬁcation and the lack of realism.”
Large-Scale Visual Relationship Understanding (2018, methodology): “Other methods either train an ofﬂine Fast/Faster-RCNN detector on VRD to extract boxes [3, 19, 38, 37, 26, 41, 35], or use the built-in RPN modules to generates proposals on-the-ﬂy [18].”
Challenges in Deploying Machine Learning: A Survey of Case Studies (2020, background): “A possible option here is to use noisy label oracles [39] or weak annotations [105], however these approaches provide imprecise labels, which can lead to a loss in quality of the model [112].”
Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation (2020, background): “Several follow-up works attempted to improve upon it: by learning a translation operator in the embedding space [43], clustering in a weakly-supervised fashion [26], using conditional random fields [8] or optimizing a cycle-consistency loss to learn object-agnostic features [39].”
MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment (2018, background): “A variety of papers have considered modeling spatial relations in natural images [7, 22, 37], and scene graph is widely used in the image retrieval tasks [24, 56].”