Pooling is neither necessary nor sufficient for appropriate deformation stability in CNNs

By Avraham Ruderman, Neil C. Rabinowitz, Ari S. Morcos, Daniel Zoran. 2018

44 citations recorded by the source corpus.

Many of our core assumptions about how neural networks operate remain empirically untested. One common assumption is that convolutional neural networks need to be stable to small translations and deformations to solve image recognition tasks. For many years, this stability was baked into CNN architectures by incorporating interleaved pooling layers. Recently, however, interleaved pooling has largely been abandoned. This raises a number of questions: Are our intuitions about deformation stability right at all? Is it important? Is pooling necessary for deformation invariance? If not, how is deformation invariance achieved in its absence? In this work, we rigorously test these questions, and find that deformation stability in convolutional networks is more nuanced than it first appears: (1) Deformation invariance is not a binary property, but rather that different tasks require different degrees of deformation stability at different layers. (2) Deformation stability is not a fixed property of a network and is heavily adjusted over the course of training, largely through the smoothness of the convolutional filters. (3) Interleaved pooling layers are neither necessary nor sufficient for achieving the optimal form of deformation stability for natural image classification. (4) Pooling confers too much deformation stability for image classification at initialization, and during training, networks have to learn to counteract this inductive bias. Together, these findings provide new insights into the role of interleaved pooling and deformation invariance in CNNs, and demonstrate the importance of rigorous empirical testing of even our most basic assumptions about the working of neural networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Materials Chemistry, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Richard Zhang

In Computer Vision and Pattern Recognition

  1. Richard Zhang
  2. Luc Van Gool
  3. Ming-Hsuan Yang
  4. Trevor Darrell
  5. Dacheng Tao
  6. Andrew Zisserman
  7. Jitendra Malik
  8. Alexei A. Efros

In Materials Chemistry

  1. Richard Zhang

In Model Architectures

  1. Richard Zhang

Citations in the corpus