WarpNet: Weakly Supervised Matching for Single-View Reconstruction

By Angjoo Kanazawa, David W. Jacobs, Manmohan Chandraker. 2016

162 citations recorded by the source corpus.

We present an approach to matching images of objects in fine-grained datasets without using part annotations, with an application to the challenging problem of weakly supervised single-view reconstruction. This is in contrast to prior works that require part annotations, since matching objects across class and pose variations is challenging with appearance features alone. We overcome this challenge through a novel deep learning architecture, WarpNet, that aligns an object in one image with a different object in another. We exploit the structure of the fine-grained dataset to create artificial data for training this network in an unsupervised-discriminative learning approach. The output of the network acts as a spatial prior that allows generalization at test time to match real images across variations in appearance, viewpoint and articulation. On the CUB-200-2011 dataset of bird categories, we improve the AP over an appearance-only network by 13.6%. We further demonstrate that our WarpNet matches, together with the structure of fine-grained datasets, allow single-view reconstructions with quality comparable to using annotated point correspondences.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “Although such covariance is beneficial for tasks such as predicting 3D correspondences [33, 57, 65], it is undesirable for most semantic recognition tasks.”
MarrNet: 3D Shape Reconstruction via 2.5D Sketches (2017, methodology): “…like ShapeNet [Chang et al., 2015], researchers developed models encoding shape prior for this task [Girdhar et al., 2016, Choy et al., 2016, Tulsiani et al., 2017, Wu et al., 2016b, Kar et al., 2015, Kanazawa et al., 2016, Soltani et al., 2017], with extension to scenes [Song et al., 2017].”
Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos (2020, methodology): “Our work builds upon prior methods in unsupervised discovery of image correspondences [23,25,21,22,8,7,11].”
Unsupervised object learning from dense equivariant image labelling (2017, methodology): “More related to our work, WarpNet [17] learns a neural network that predicts a Thin Plate Spline transformations between pairs of image of an object.”
Unsupervised End-to-end Learning for Deformable Medical Image Registration (2017, background): “Several STN-based approaches have been proposed to address similar problems in natural scenes, such as optical flow estimation [23], [24] depth estimation [25], [26] and single-view reconstruction [27].”
DSAC — Differentiable RANSAC for Camera Localization (2016, background): “Recently, deep learning has been shown to be highly successful at image recognition tasks [37, 17, 13, 31], and, increasingly, in other domains including geometry [10, 19, 20, 9].”
Convolutional Neural Network Architecture for Geometric Matching (2017, methodology): “Re-placing our correlation-based matching layer with feature concatenation or subtraction, as proposed in [14] and [30], respectively, incurs a large performance drop.”
Recycle-GAN: Unsupervised Video Retargeting (2018, background): “Learning Association: Much of computer vision is about learning association, be it learning high-level image classiﬁcation [38], object relationships [32], or point-wise correspondences [2,24,29,31].”
GAN-Supervised Dense Visual Alignment (2021, background): “, classification) and applications such as robust filter learning [16, 36], view synthesis [26, 63, 90] and 3D representation learning [38, 83, 88].”
AutoScaler: Scale-Attention Networks for Visual Correspondence (2016, methodology): “The competing algorithms include CNN-31x31, a nine-layer fully convolutional network used in [34].”
W-TALC: Weakly-supervised Temporal Activity Localization and Classification (2018, background): “In computer vision, researchers have utilized weak labels to learn models for several tasks including semantic segmentation [18,28,63], visual tracking [69], reconstruction [52,25], video summarization [37], learning robotic manipulations [46], video captioning [41], object boundaries [29], place re”
VITON: An Image-Based Virtual Try-on Network (2017, background): “Note that a potential alternative to estimating TPS with shape context matching is to learn TPS parameters through a Siamese network as in [23].”
Unsupervised Learning of Object Landmarks by Factorized Spatial Embeddings (2017, methodology): “Instead, we randomly sample two TPS transformations ( g 1 , g 2 ) and consecutively warp the given image to generate an image pair i.e . x = I ◦ g 1 and x (cid:48) = x ◦ g 2 (computed us-ing inverse image warping as x ◦ ( g 2 ◦ g 1 ) ).”
Unsupervised Discovery of Object Landmarks as Structural Representations (2018, background): “WarpNet [24] corresponds images in the same class by estimating the parameter of a thin plate spline (TPS) transformation [4], and it can roughly reconstruct 3D point cloud using a single-view image.”
Universal Correspondence Network (2016, methodology): “Figure 8: PCK on CUB dataset [31], compared with various other approaches including WarpNet [16] (L = √ w2 + h2.”
Semantic Video Segmentation by Gated Recurrent Flow Propagation (2016, methodology): “d we can backpropagate gradients from yto both xand f. The sum contains only 4 non-zero terms when using the bilinear kernel, so it can be computed efﬁciently. A similar modiﬁ- cation was proposed in [16] for single-view reconstruction. 3.2.GatedRecurrentUnitsforSemanticVideoSegmentation To connect ”
Detail-Revealing Deep Video Super-Resolution (2017, methodology): “Based on it, WarpNet [13] used a similar scheme to extract sparse correspondence.”
Self-Supervised Learning of Geometrically Stable Features Through Probabilistic Introspection (2018, methodology): “Artificial correspondences were also used in [15] for fine-grained categories.”