An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution

By Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alexander Sergeev, Jason Yosinski. 2018

1,041 citations recorded by the source corpus.

Few ideas have enjoyed as large an impact on deep learning as convolution. For any problem involving pixels or spatial representations, common intuition holds that convolutional neural networks may be appropriate. In this paper we show a striking counterexample to this intuition via the seemingly trivial coordinate transform problem, which simply requires learning a mapping between coordinates in (x,y) Cartesian space and one-hot pixel space. Although convolutional networks would seem appropriate for this task, we show that they fail spectacularly. We demonstrate and carefully analyze the failure first on a toy problem, at which point a simple fix becomes obvious. We call this solution CoordConv, which works by giving convolution access to its own input coordinates through the use of extra coordinate channels. Without sacrificing the computational and parametric efficiency of ordinary convolution, CoordConv allows networks to learn either complete translation invariance or varying degrees of translation dependence, as required by the end task. CoordConv solves the coordinate transform problem with perfect generalization and 150 times faster with 10--100 times fewer parameters than convolution. This stark contrast raises the question: to what extent has this inability of convolution persisted insidiously inside other tasks, subtly hampering performance from within? A complete answer to this question will require further investigation, but we show preliminary evidence that swapping convolution for CoordConv can improve models on a diverse set of tasks. Using CoordConv in a GAN produced less mode collapse as the transform between high-level spatial latents and pixels becomes easier to learn. A Faster R-CNN detection model trained on MNIST showed 24% better IOU when using CoordConv, and in the RL domain agents playing Atari games benefit significantly from the use of CoordConv layers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Exploring Self-Attention for Image Recognition (2020, methodology): “Absolute position encoding [20] is better Method top-1 top-5 Params Flops”
Attention Augmented Convolutional Networks (2019, methodology): “In particular, the Image Transformer [32] extends the sinusoidal waves first introduced in the original Transformer [43] to 2 dimensional inputs and CoordConv [29] concatenates positional channels to an activation map.”
Global Pooling, More than Meets the Eye: Position Information is Encoded Channel-Wise in CNNs (2021, background): “The ability of zero padding to inject position information has been an area of focus in multiple follow-up works [18, 1, 16, 23, 24].”
Implicit Behavioral Cloning (2021, background): “4 shows how on a simple visual coordinate regression task, which is a notoriously hard problem84 for convolutional networks [23], an MSE-trained Conv-MLP model [24] with CoordConv [23] struggles85 to extrapolate outside the convex hull of the training data.”
Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs (2019, methodology): “The idea of appending coordinate channels to convolutional layers has recently been highlighted (and named CoordConv) in the context of improving positional generalization [Liu et al., 2018].”
COCO-GAN: Generation by Parts via Conditional Coordinating (2019, methodology): “CoordConv [15] is another similar method but with fundamental differences.”
Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth (2019, background): “Recently, works [19, 14] have shown that the spatial-invariant nature of Fully Convolutional Networks is not ideal for embedding methods and propose to either incorporate coordinate maps [14] or use so-called semiconvolutions [19] to alleviate this problem.”
AdaptIS: Adaptive Instance Selection Network (2019, background): “[22] proposed a simple CoordConv layer that feeds pixel coordinates to the network.”
MoViNets: Mobile Video Networks for Efficient Video Recognition (2021, methodology): “To combat this problem, we apply a sine-based fixed positional encoding (POSENC) scheme inspired by Transformers [66, 42].”
InfinityGAN: Towards Infinite-Resolution Image Synthesis (2021, methodology): “In practice, as the grid-shaped z l and c are sampled with equal spacing between consecutive pixels, the feature unfolding can be efﬁciently implemented with CoordConv [20].”
Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression (2019, methodology): “[38], we encode into our model the full coordinate information and the information only on boundaries predicted from the previous HG module into our model.”
PENet: Towards Precise and Efficient Image Guided Depth Completion (2021, methodology): “Our experiments show that the proposed geometric convolutional layer can considerably improve the depth completion performance but CoordConv [22] is not helpful.”
The CoSTAR Block Stacking Dataset: Learning with Workspace Constraints (2018, methodology): “CoordConv [41] “Pre-Image” applies an initial CoordConv Layer to each input image and CoordConv “Pre-Trunk” applies a CoordConv layer after the vision and vector branches have been concatenated in the HyperTree Trunk.”
Structural Autoencoders Improve Representations for Generation and Transfer (2020, methodology): “The input image is split into overlapping 4x4 patches (represented with red boundaries in Figure 1a) each of which are processed independently, with the corresponding position coordinates of the patch concatenated to break the translation invariance as in [20].”
Do We Really Need Explicit Position Encodings for Vision Transformers? (2021, neutral): “Coord-Conv [26] uses concatenation instead of addition.”
Semi-convolutional Operators for Instance Segmentation (2018, background): “interesting concurrent work that explores related convolutional dilemmas [36]).”