Making Convolutional Networks Shift-Invariant Again

By Richard Zhang. 2019

969 citations recorded by the source corpus.

Modern convolutional networks are not shift-invariant, as small input shifts or translations can cause drastic changes in the output. Commonly used downsampling methods, such as max-pooling, strided-convolution, and average-pooling, ignore the sampling theorem. The well-known signal processing fix is anti-aliasing by low-pass filtering before downsampling. However, simply inserting this module into deep networks degrades performance; as a result, it is seldomly used today. We show that when integrated correctly, it is compatible with existing architectural components, such as max-pooling and strided-convolution. We observe \textit{increased accuracy} in ImageNet classification, across several commonly-used architectures, such as ResNet, DenseNet, and MobileNet, indicating effective regularization. Furthermore, we observe \textit{better generalization}, in terms of stability and robustness to input corruptions. Our results demonstrate that this classical signal processing technique has been undeservingly overlooked in modern deep networks. Code and anti-aliased versions of popular networks are available at this https URL .

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

On Robustness and Transferability of Convolutional Neural Networks (2020, background): “Subsequent studies have investigated brittleness to similar corruptions [53, 76], as well as to impulse noise [31], photometric perturbations [62], and small shifts and other transformations [2, 17, 74].”
Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, background): “Translation equivariance, although not exact [97], aligns well with the nature of imaging and thus generalizes the model to different positions or to images of different sizes.”
Self-Training With Noisy Student Improves ImageNet Classification (2019, background): “0 Low Pass Filter Pooling [92] 224 - 51.”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, methodology): “Encouragingly, the past few years have seen substantial progress in robustness to these distribution shifts, e.g., see [13, 28, 34, 55, 57, 66, 93, 96, 105, 114, 115] among many others.”
Towards Robust Vision Transformer (2021, background): “, stronger data augmentation [16, 18, 33], carefully designed [36,44] or searched [9,13] network architecture, improved training strategy [22, 26, 47], quantization [24] and pruning [49] of the weights, better pooling [41, 53] or activation functions [46], etc.”
CNN-Generated Images Are Surprisingly Easy to Spot… for Now (2019, background): “Azulay and Weiss [6] showed convolutional networks ignore the classical sampling theorem and that strided convolutions therefore reduce translation invariance, and Zhang [49] improved translation invariance by reducing aliasing in these layers.”
Analyzing and Improving the Image Quality of StyleGAN (2019, neutral): “We believe the problem is that in progressive growing each resolution serves momentarily as the output resolution, forcing it to generate maximal frequency details, which then leads to the trained network to have excessively high frequencies in the intermediate layers, compromising shift invariance ”
A Style-Based Generator Architecture for Generative Adversarial Networks (2018, methodology): “We first switch to an improved baseline (b) by using bilinear up/downsampling operations [58], longer training, and tuned hyperparameters.”
Score-based Generative Modeling in Latent Space (2021, methodology): “Some of our models use upsampling and downsampling operations with anti-aliasing based on Finite Impulse Response (FIR) [107], following Song et al. [2].”
On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, background): “While it was replaced by operations such as max-pooling, based on empirical performance [65], recent works have demonstrated that antialiasing can be compatible and improve performance in convolutional networks [79,85], transformers [57], NeRFs [4], and GANs [36].”
Global Pooling, More than Meets the Eye: Position Information is Encoded Channel-Wise in CNNs (2021, neutral): “We adopt the same consistency (Cons.) metric as proposed previously [34] to measure the translation invariance.”
AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty (2019, background): “MaxBlur Pooling (Zhang, 2019) is a recently proposed architectural modification which smooths the results of pooling.”
Why do deep convolutional networks generalize so poorly to small image transformations? (2018, background): “In the recent work of (Zhang, 2019), an elegant proposal was made regarding how to embody this principle in modern CNNs. Speciﬁcally, they proposed performing the pooling operation in two steps: a max-pool layer with stride 1, followed by a convolutional layer with stride=2.”
Toward Causal Representation Learning (2021, background): “In computer vision [76], [228], changes in the test distribution may, for instance, come from aberrations, such as camera blur, noise, or compression quality [107], [129], [170], [206], or from shifts, rotations, or viewpoints [7], [12], [64], [282].”
Towards Causal Representation Learning (2021, background): “In computer vision [75, 228], changes in the test distribution may, for instance, come from aberrations like camera blur, noise or compression quality [106, 129, 170, 206], or from shifts, rotations, or viewpoints [7, 11, 63, 282].”