An Effective Anti-Aliasing Approach for Residual Networks

By Cristina Nader Vasconcelos, Hugo Larochelle, Vincent Dumoulin, Nicolas Le Roux, Ross Goroshin. 2020

36 citations recorded by the source corpus.

Image pre-processing in the frequency domain has traditionally played a vital role in computer vision and was even part of the standard pipeline in the early days of deep learning. However, with the advent of large datasets, many practitioners concluded that this was unnecessary due to the belief that these priors can be learned from the data itself. Frequency aliasing is a phenomenon that may occur when sub-sampling any signal, such as an image or feature map, causing distortion in the sub-sampled output. We show that we can mitigate this effect by placing non-trainable blur filters and using smooth activation functions at key locations, particularly where networks lack the capacity to learn them. These simple architectural changes lead to substantial improvements in out-of-distribution generalization on both image classification under natural corruptions on ImageNet-C [10] and few-shot learning on Meta-Dataset [17], without introducing additional trainable parameters and using the default hyper-parameters of open source codebases.

Subfields: Artificial Intelligence, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

Citations in the corpus

Towards Robust Vision Transformer (2021, background): “, stronger data augmentation [16, 18, 33], carefully designed [36,44] or searched [9,13] network architecture, improved training strategy [22, 26, 47], quantization [24] and pruning [49] of the weights, better pooling [41, 53] or activation functions [46], etc.”
Alias-Free Generative Adversarial Networks (2021, background): “This holds for all filters commonly used in deep learning [65, 57], and even high-quality filters used in image processing.”
MedViT: A Robust Vision Transformer for Generalized Medical Image Classification (2023, background): “36], strong data augmentation [37, 38], searched network architecture [39, 40], improved training strategy [41–43], pruning [44] of the weights, and quantization [45], activation functions [46] or better pooling [47, 48].”
Blending Anti-Aliasing into Vision Transformer (2021, background): “These anti-aliased CNNs demonstrate both better recognition, generalization to downstream tasks and robustness towards corruptions [37, 38].”
Improving Sound Event Classification by Increasing Shift Invariance in Convolutional Neural Networks (2021, methodology): “The ﬁndings above have led to the rise of a recent and growing area of research aimed at increasing shift invariance in CNNs, either through architectural improvements [7, 9, 10] or via data augmentation [6].”