DeepSetNet: Predicting Sets with Deep Neural Networks

By Seyed Hamid Rezatofighi, B. V. Kumar, Anton Milan, Ehsan Abbasnejad, Anthony R. Dick, Ian David Reid. 2016

56 citations recorded by the source corpus.

This paper addresses the task of set prediction using deep learning. This is important because the output of many computer vision tasks, including image tagging and object detection, are naturally expressed as sets of entities rather than vectors. As opposed to a vector, the size of a set is not fixed in advance, and it is invariant to the ordering of entities within it. We define a likelihood for a set distribution and learn its parameters using a deep neural network. We also derive a loss for predicting a discrete distribution corresponding to set cardinality. Set prediction is demonstrated on the problem of multi-class image classification. Moreover, we show that the proposed cardinality loss can also trivially be applied to the tasks of object counting and pedestrian detection. Our approach outperforms existing methods in all three cases on standard datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Media Technology, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Media Technology

In Model Architectures

Citations in the corpus

End-to-End Object Detection with Transformers (2020, background): “The basic set prediction task is multilabel classiﬁcation (see e.g., [40,33] for references in the context of computer vision) for which the baseline approach, one-vs-rest, does not apply to problems such as detection where there is an underlying structure between elements (i.e., near-identical…”
Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, background): “2017 [51] VGG 3 74505 8278 40504 62.”
Deep Perm-Set Net: Learn to predict sets with unknown permutation and cardinality using deep neural networks (2018, background): “Set learning using deep networks is an emerging field of study that has generated substantial interest very recently Rezatofighi et al. (2017; 2018); Vinyals et al. (2015); Zaheer et al. (2017).”
Set Prediction without Imposing Structure as Conditional Density Estimation (2020, methodology): “This sets the approach apart from forward-processing models, that either require conditional independence assumptions (Rezatoﬁghi et al., 2017), or an order on the predictions, when applying the chain rule (Vinyals et al., 2016).”
Loss Functions for Multiset Prediction (2017, background): “Representative approaches include binary relevance, which assumes classes are conditionally independent, and probabilistic classifier chains which decompose the joint probability as p(y|x) = ∏|C| c=1 p(yc|y<c, x) [3, 17, 22, 7].”
Inverse Cooking: Recipe Generation From Food Images (2018, background): “As an alternative, researchers have attempted to predict the cardinality of the set of labels [43, 44]; however, assuming the independence of labels.”
Learn to Predict Sets Using Feed-Forward Neural Networks (2020, methodology): “Set learning using deep networks has generated substantial interest very recently [7], [8], [9], [10], [11], [12], [13].”
FSPool: Learning Set Representations with Featurewise Sort Pooling (2019, methodology): “(2018); Rezatofighi. et al. (2017).”
Joint Learning of Set Cardinality and State Distribution (2017, methodology): “We follow the same experimental setup used in (Rezatoﬁghi et al. 2017; Wang et al. 2016).”