Recombinator Networks: Learning Coarse-to-Fine Feature Aggregation

By Sina Honari, Jason Yosinski, Pascal Vincent, C. Pal. 2015

127 citations recorded by the source corpus.

Deep neural networks with alternating convolutional, max-pooling and decimation layers are widely used in state of the art architectures for computer vision. Max-pooling purposefully discards precise spatial information in order to create features that are more robust, and typically organized as lower resolution spatial feature maps. On some tasks, such as whole-image classification, max-pooling derived features are well suited, however, for tasks requiring precise localization, such as pixel level prediction and segmentation, max-pooling destroys exactly the information required to perform well. Precise localization may be preserved by shallow convnets without pooling but at the expense of robustness. Can we have our max-pooled multilayered cake and eat it too? Several papers have proposed summation and concatenation based methods for combining upsampled coarse, abstract features with finer features to produce robust pixel level predictions. Here we introduce another model - dubbed Recombinator Networks - where coarse features inform finer features early in their formation such that finer features can make use of several layers of computation in deciding how to use coarse features. The model is trained once, end-to-end and performs better than summation-based architectures, reducing the error from the previous state of the art on two facial keypoint datasets, AFW and AFLW, by 30% and beating the current state-of-the-art on 300W without using extra data. We improve performance even further by adding a denoising prediction model based on a novel convnet formulation.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Feature Pyramid Networks for Object Detection (2016, methodology): “There are recent methods exploiting lateral/skip connections that associate low-level feature maps across resolutions and semantic levels, including U-Net [31] and SharpMask [28] for segmentation, Recombinator networks [17] for face detection, and Stacked Hourglass networks [26] for keypoint estimat”
Panoptic Feature Pyramid Networks (2019, methodology): “As an alternative to dilation, an encoder-decoder [2] or ‘U-Net’ [49] architecture can be used to increase feature resolution [26, 44, 20, 47].”
Convolutional Neural Fabrics (2016, methodology): “Finally, we analyze the number of response maps, parameters, and activations of fabrics in Section 3.3.”
Deep High-Resolution Representation Learning for Visual Recognition (2019, background): “…with dilated convolutions used in the back-bone [63], [89], [113]; light downsample and heavy up-sample processes [141], recombinator networks [55]; improving skip connections with more or complicated convolutional units [64], [111], [180], as well as sending information from low-resolution…”
MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation (2022, methodology): “For our denoising network we use a U-net architecture [Ronneberger et al., 2015, Honari et al., 2016, Salimans et al., 2017] combining the improvements from Song et al. [2021] and Dhariwal and Nichol [2021].”
Towards Multi-class Object Detection in Unconstrained Remote Sensing Imagery (2018, background): “Another alternative is image or feature pyramids [7, 12, 14, 20, 21, 31].”
Automatic Analysis of Facial Actions: A Survey (2019, background): “The response map creation and the shape fitting can then both be combined into an end-to-end training [78].”
An Architecture for Deep, Hierarchical Generative Models (2016, methodology): “This approach is exempliﬁed by the Ladder Network of Rasmus et al. [17], and has been developed further for, e.g. generative modelling and image processing in [8, 23].”
ELASTIC: Improving CNNs With Dynamic Scaling Policies (2018, methodology): “Several recent methods including SharpMask [27] and U-Net [28] for segmentation, Stacked Hourglass networks [26] for keypoint estimation and Recombinator networks [11] for face detection, have used skip connections to incorporate low-level feature maps on multiple resolutions and semantic levels.”
High-Resolution Representations for Labeling Pixels and Regions (2019, unknown): “RCN [40] 5.”
Whole-Body Human Pose Estimation in the Wild (2020, methodology): “RCN [18] - 4.”
Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals (2017, methodology): “Overall, RCN3D track is more accurate than THU VCLab track, which itself outperforms NAIST RV track.”