Learn to Predict Sets Using Feed-Forward Neural Networks

By Hamid Rezatofighi, Tianyu Zhu, Roman Kaskman, Farbod T. Motlagh, Javen Qinfeng Shi, Anton Milan, Daniel Cremers, L. Leal-Taix'e, Ian David Reid. 2020

21 citations recorded by the source corpus.

This paper addresses the task of set prediction using deep feed-forward neural networks. A set is a collection of elements which is invariant under permutation and the size of a set is not fixed in advance. Many real-world problems, such as image tagging and object detection, have outputs that are naturally expressed as sets of entities. This creates a challenge for traditional deep neural networks which naturally deal with structured outputs such as vectors, matrices or tensors. We present a novel approach for learning to predict sets with unknown permutation and cardinality using deep neural networks. In our formulation we define a likelihood for a set distribution represented by a) two discrete distributions defining the set cardinally and permutation variables, and b) a joint distribution over set elements with a fixed cardinality. Depending on the problem under consideration, we define different training models for set prediction using deep neural networks. We demonstrate the validity of our set formulations on relevant vision problems such as: 1) multi-label image classification where we outperform the other competing methods on the PASCAL VOC and MS COCO datasets, 2) object detection, for which our formulation outperforms popular state-of-the-art detectors, and 3) a complex CAPTCHA test, where we observe that, surprisingly, our set-based network acquired the ability of mimicking arithmetics without any rules being coded.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, background): “Neural networks for sets A range of recent methods explore set encoding [34, 50, 51], generation [31, 52], and set-to-set mappings [20, 53].”
Set Prediction without Imposing Structure as Conditional Density Estimation (2020, methodology): “In this work we focus on distributions taking the form of deep energy-based models (Ngiam et al., 2011; Zhai et al., 2016; Belanger & McCallum, 2016): with Z as the partition function and E θ the energy function with parameters θ .”