On Translation Invariance in CNNs: Convolutional Layers Can Exploit Absolute Spatial Location

By Osman Semih Kayhan, Jan van Gemert. 2020

278 citations recorded by the source corpus.

In this paper we challenge the common assumption that convolutional layers in modern CNNs are translation invariant. We show that CNNs can and will exploit the absolute spatial location by learning filters that respond exclusively to particular absolute locations by exploiting image boundary effects. Because modern CNNs filters have a huge receptive field, these boundary effects operate even far from the image boundary, allowing the network to exploit absolute spatial location all over the image. We give a simple solution to remove spatial location encoding which improves translation invariance and thus gives a stronger visual inductive bias which particularly benefits small data sets. We broadly demonstrate these benefits on several architectures and various applications such as image classification, patch matching, and two video classification datasets.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Global Pooling, More than Meets the Eye: Position Information is Encoded Channel-Wise in CNNs (2021, neutral): “Recent works [14, 18, 1, 16] showed that CNNs exploit absolute position information.”
Alias-Free Generative Adversarial Networks (2021, background): “known to leak absolute image coordinates into the internal representations [27, 34, 63].”
Hallucination In Object Detection — A Study In Visual Part VERIFICATION (2021, background): “Object hallucination by deep detectors can be causes by sensitivity to the absolute position in the image [14, 15] while also affected by scene context [16, 17, 18, 19, 20].”
CMT: Convolutional Neural Networks Meet Vision Transformers (2021, unknown): “In other words, we expect translation-invariance [27] in those tasks.”
StyleSwin: Transformer-based GAN for High-resolution Image Generation (2021, methodology): “However, when substituting the convolutional layers with transformers that use RPE, one thing is rarely noticed: ConvNets could infer the absolute positions by leveraging the clues from the zero paddings [22,31] yet such feature is missing in Swin blocks using RPE.”
Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs (2022, background): “Large kernels not only help to learn the relative positions between concepts, but also encode the absolute position information due to padding effect [53].”
High-fidelity GAN Inversion with Padding Space (2022, background): “side effect of constant padding used in CNNs has recently been studied [3, 18, 19, 23, 26, 45].”
MPViT: Multi-Path Vision Transformer for Dense Prediction (2021, background): “On the contrary, CNNs can exploit local connectivity from translation invariance [31, 52] – each patch in an image is processed by the same weights.”
Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D (2020, background): “Fully convolutional singleimage object detectors roughly have this property and the multi-view extension inherits this property from them [11] [6].”
Mind the Pad - CNNs can Develop Blind Spots (2020, background): “Zero padding has been recently linked to CNNs’ ability to encode position information [6; 14; 16].”
TS2Vec: Towards Universal Representation of Time Series (2021, background): “It has been proved that convolutional layers can implicitly encode positional information into into latent space (Islam, Jia, and Bruce 2019; Kayhan and Gemert 2020).”
Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer (2022, background): “When a short video clip is encoded, the zero paddings in the temporal dimension also get encoded and affect the output tokens, causing an unbalanced effects to tokens at different temporal position [24,33,80,3].”