Learning Prototypes for Visual Relationship Detection

By François Plesse, Alexandru-Lucian Gînsca, Bertrand Delezoide, Françoise J. Prêteux. 2018

7 citations recorded by the source corpus.

Relationships between objects drive our understanding of images, but modelling them poses several challenges due to the combinatorial nature of the problem and the complex structure of natural language. This paper tackles the task of predicting relationships in the form of (subject, predicate, object) triplets from still images. To address these issues, we propose a framework for learning predicate prototypes that aims to capture the multimodal nature of predicate distributions. Concurrently, a network is trained to define a space in which relationships with similar spatial layouts, interacting objects and predicates are clustered together. Finally, at test time, relationships are classified based on the similarity to their train nearest neighbor. We evaluate this approach on two well known datasets and observe comparable performance with state of the art models. Furthermore, we find that coupling prototype learning with a nearest neighbors approach increases the performance from 85.4 % to 87.6 % over a standard classification approach.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

This Looks Like That, Because ... Explaining Prototypes for Interpretable Image Recognition (2020, background): “, images from the training set that look similar to the image being classified [2, 6, 27, 32].”