Deep Perm-Set Net: Learn to predict sets with unknown permutation and cardinality using deep neural networks

By Seyed Hamid Rezatofighi, Roman Kaskman, Farbod T. Motlagh, Javen Qinfeng Shi, Daniel Cremers, L. Leal-Taix'e, Ian David Reid. 2018

26 citations recorded by the source corpus.

Many real-world problems, e.g. object detection, have outputs that are naturally expressed as sets of entities. This creates a challenge for traditional deep neural networks which naturally deal with structured outputs such as vectors, matrices or tensors. We present a novel approach for learning to predict sets with unknown permutation and cardinality using deep neural networks. Specifically, in our formulation we incorporate the permutation as unobservable variable and estimate its distribution during the learning process using alternating optimization. We demonstrate the validity of this new formulation on two relevant vision problems: object detection, for which our formulation outperforms state-of-the-art detectors such as Faster R-CNN and YOLO, and a complex CAPTCHA test, where we observe that, surprisingly, our set based network acquired the ability of mimicking arithmetics without any rules being coded.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

End-to-End Object Detection with Transformers (2020, background): “…advances in complex structured prediction tasks such as machine translation or speech recognition, but not yet in object detection: previous attempts [43,16,4,39] either add other forms of prior knowledge, or have not proven to be competitive with strong baselines on challenging benchmarks.”
Learn to Predict Sets Using Feed-Forward Neural Networks (2020, methodology): “Following our 2018 Arxiv work [28], two very recent frameworks [29], [30] have adopted this new re-formulation, i.”
FSPool: Learning Set Representations with Featurewise Sort Pooling (2019, methodology): “An MLP or RNN that tries to generate these (like in Rezatoﬁghi et al. (2018); Stewart & Andriluka (2016)) must handle which of its outputs is responsible for what element in a discontinuous way.”
Set-Structured Latent Representations (2020, methodology): “, several multi-layer perceptrons) and then use the elements of the collection in an arbitrary order (Stewart et al., 2015; Rezatofighi et al., 2018).”
MetaAnchor: Learning to Detect Objects with Customized Anchors (2018, background): “anchor box choices and the definition of (1)There are a few recent studies on the topic, such as [33, 37].”
Deep Set Prediction Networks (2019, methodology): “This has been used for predicting point clouds [1; 5], bounding boxes [15; 2], and graphs (sets of nodes and edges) [3; 17].”
Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs (2018, methodology): “Recently, Rezatoﬁghi et al. (2018) proposed a method that computes the posterior distribution of all permutations, conditioned on the model and the data.”