Scaling the Scattering Transform: Deep Hybrid Networks

By Edouard Oyallon, Eugene Belilovsky, Sergey Zagoruyko. 2017

171 citations recorded by the source corpus.

We use the scattering network as a generic and fixed initialization of the first layers of a supervised hybrid deep network. We show that early layers do not necessarily need to be learned, providing the best results to-date with pre-defined representations while being competitive with Deep CNNs. Using a shallow cascade of 1 × 1 convolutions, which encodes scattering coefficients that correspond to spatial windows of very small sizes, permits to obtain AlexNet accuracy on the imagenet ILSVRC2012. We demonstrate that this local encoding explicitly learns invariance w.r.t. rotations. Combining scattering networks with a modern ResNet, we achieve a single-crop top 5 error of 11.4% on imagenet ILSVRC2012, comparable to the Resnet-18 architecture, while utilizing only 10 layers. We also find that hybrid architectures can yield excellent performance in the small sample regime, exceeding their end-to-end counterparts, through their ability to incorporate geometrical priors. We demonstrate this on subsets of the CIFAR-10 dataset and on the STL-10 dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Ocean Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Ocean Engineering

In Model Architectures

Citations in the corpus

A critical analysis of self-supervision, or what we can learn from a single image (2019, methodology): “Notably, these models also outperform Gabor-like filters from Scattering networks (Bruna & Mallat, 2013), which are hand crafted image features, replacing the first two convolutional layers as in (Oyallon et al., 2017).”
Unsupervised Embedding Learning via Invariant and Spreading Instance Feature (2019, neutral): “Those results are taken from [33].”
Invariant Information Clustering for Unsupervised Image Classification and Segmentation (2018, neutral): “3 Oyallon* 2017 [42]† 76.”
Scattering Networks for Hybrid Representation Learning (2018, background): “This article is an extended version of [40].”
Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet (2019, methodology): “Scattering networks Another related work by Oyallon et al. (2017) uses a scattering network with small receptive ﬁelds (14 x 14 pixels) in conjunction with a two-layer Multilayer Perceptron or a ResNet-10 on top of the scattering network.”
AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations Rather Than Data (2019, methodology): “It is worth pointing out on CIFAR-10, the other reported methods [21, 7, 25, 20, 10] are usually based on different unsupervised networks and supervised classifiers for evaluation, making it difficult to make a direct comparison between them.”
Small Data Challenges in Big Data Era: A Survey of Recent Progress on Unsupervised and Semi-Supervised Methods (2019, unknown): “Although not all approaches have been compared against this protocol due to the legacy reason, it has emerged in literature to allow a fair and direct comparison among many recent methods [9], [37], [39], [41], [76], [77], [78].”
Whitening for Self-Supervised Representation Learning (2020, methodology): “Fully and semi-supervised classication on STL10. a indicates results reported in [26]. yindicates fully supervised method. Method Accuracy Dundar et al. [13] a 74.1 Cutout [9] ya 87.3 Oyallon et al. [34] ya 87.6 DeepCluster [3] a 73.4 ADC [18] a 56.7 DIM [23] a 77.0 IIC [26] a 88.8 InfoNCE 87.09 W-M”
On Regularization and Robustness of Deep Neural Networks (2018, background): “While other approaches may also be important to solve this grand challenge, such as incorporating more prior knowledge in the architecture (e.g., Oyallon et al., 2017), semi-supervised learning (Chapelle et al., 2006) or meta-learning (when multiple tasks or datasets are available, see, e.g.,…”
Shallow Learning For Deep Networks (2018, methodology): “In addition to exceeding AlexNet, this model compares favorably to all alternatives to end-to-end supervised CNNs including hand crafted computer vision and unsupervised learning techniques (Noroozi & Favaro, 2016; Perronnin & Larlus, 2015; Sánchez et al., 2013; Oyallon et al., 2017) (full results s”
Steering Self-Supervised Feature Learning Beyond Local Pixel Statistics (2020, unknown): “[51] 87.6% Swersky et al .”