DeepPermNet: Visual Permutation Learning

By Rodrigo Santa Cruz, Basura Fernando, Anoop Cherian, Stephen Gould. 2017

109 citations recorded by the source corpus.

We present a principled approach to uncover the structure of visual data by solving a novel deep learning task coined visual permutation learning. The goal of this task is to find the permutation that recovers the structure of data from shuffled versions of it. In the case of natural images, this task boils down to recovering the original image from patches shuffled by an unknown permutation matrix. Unfortunately, permutation matrices are discrete, thereby posing difficulties for gradient-based methods. To this end, we resort to a continuous approximation of these matrices using doubly-stochastic matrices which we generate from standard CNN predictions using Sinkhorn iterations. Unrolling these iterations in a Sinkhorn network layer, we propose DeepPermNet, an end-to-end CNN model for this task. The utility of DeepPermNet is demonstrated on two challenging computer vision problems, namely, (i) relative attributes learning and (ii) self-supervised representation learning. Our results show state-of-the-art performance on the Public Figures and OSR benchmarks for (i) and on the classification and segmentation tasks on the PASCAL VOC dataset for (ii).

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Invariant Information Clustering for Unsupervised Image Classification and Segmentation (2018, background): “The latter includes objectives such as generative autoencoder image reconstruction [46], triplets [44] and spatial-temporal order or context prediction [36, 12, 16], for example predicting patch proximity [29], solving jig-saw puzzles [40] and inpainting [41].”
PointCNN: Convolution On X-Transformed Points (2018, background): “This is different than that in [8], where a permutation matrix is the desired output, and is approximated by a doubly stochastic matrix.”
Temporal Cycle-Consistency Learning (2019, background): “…order [16, 27], consistency across viewpoints and/or temporal neighbors [35], classifying arbitrary temporal segments [22], temporal distance classiﬁ-cation within or across modalities [7], spatial permutation of patches [5, 14], visual similarity [32] or a combination of such signals [15].”
Geographical Knowledge-Driven Representation Learning for Remote Sensing Images (2021, background): “rearrangement of patches, from which image representations are learned [32, 33, 36].”
RPM-Net: Robust Point Matching Using Learned Features (2020, methodology): “We follow [29] in our implementation of the Sinkhorn normalization by unrolling it for a fixed number of steps (set to 5).”
Improving Spatiotemporal Self-Supervision by Deep Reinforcement Learning (2018, background): “It has been pursued on still images [34, 36, 10, 9, 11] by employing spatial shuﬄing of images (especially permuting jigsaws) and in videos [32, 28, 17, 6] by utilizing temporally shuﬄed sequences.”
Learning Latent Permutations with Gumbel-Sinkhorn Networks (2018, methodology): “A more complex scenario for learning permutations arises in the reconstruction of an image X from a collection of scrambled “jigsaw” pieces ˜ X (Noroozi & Favaro, 2016; Cruz et al., 2017).”