Understanding the Effective Receptive Field in Deep Convolutional Neural Networks

By Wenjie Luo, Yujia Li, Raquel Urtasun, Richard S. Zemel. 2016

2,267 citations recorded by the source corpus.

We study characteristics of receptive fields of units in deep convolutional networks. The receptive field size is a crucial issue in many visual tasks, as the output must respond to large enough areas in the image to capture information about large objects. We introduce the notion of an effective receptive field size, and show that it both has a Gaussian distribution and only occupies a fraction of the full theoretical receptive field size. We analyze the effective receptive field in several architecture designs, and the effect of sub-sampling, skip connections, dropout and nonlinear activations on it. This leads to suggestions for ways to address its tendency to be too small.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, background): “Although the receptive field might cover the whole clip, the effective receptive field has a Gaussian distribution [20], producing features focusing on different temporal regions of a clip.”
Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, background): “A large effective receptive field [29] is essential for understanding the global structure of an image and hence solving the inpainting problem.”
Deformable Convolutional Networks (2017, background): “Effective Receptive Field [43] It ﬁnds that not all pixels in a receptive ﬁeld contribute equally to an output response.”
Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks (2018, background): “However, the effective size of such fields is in practice considerably smaller [27].”
StairNet: Top-Down Semantic Aggregation for Accurate One Shot Detection (2017, background): “[24] shows that the pixels near the center of receptive field have much larger effect than the outer pixels, leading to a 2D-gaussian shape that also occupies smaller fraction than the trf.”
CopulaNet: Learning residue co-evolution directly from multiple sequence alignment for protein structure prediction (2020, background): “relationship, it focuses on neighboring residues rather than distant pairs [25].”
Transporter Networks: Rearranging the Visual World for Robotic Manipulation (2020, background): “To enable this, it is important to increase the receptive field [38] of the FCNs to encompass most of the visual observation ot.”
Deformable ConvNets V2: More Deformable, Better Results (2018, methodology): “To better understand the behavior of Deformable ConvNets, we visualize the spatial support of network nodes by their effective receptive fields [30], effective sampling locations, and error-bounded saliency regions.”
MLP-Mixer: An all-MLP Architecture for Vision (2021, background): “Neurons in deeper layers have a larger receptive field [1, 29].”
Scale-Aware Trident Networks for Object Detection (2019, background): “This finding shares the same spirit in [31] that the effective receptive field is smaller than the theoretical receptive field.”
Video Frame Interpolation via Adaptive Separable Convolution (2017, background): “Pooling layers have a great in-ﬂuence on the receptive ﬁeld [32] of a convolutional neural network, which in our context is related to the aperture problem in motion estimation.”
Learning with Limited Annotations: A Survey on Deep Semi-Supervised Learning for Medical Image Segmentation (2022, methodology): “can enlarge foreground features while skip-connections will shrink foreground features [42], [43].”
Improving Semantic Segmentation via Decoupled Body and Edge Supervision (2020, background): “First, the Receptive Field (RF) of FCNs grows slowly (only linearly) with increasing depth in the network, and such the limited RF is not able to fully model the longer-range relationships between pixels in an image [4,5].”
Do Vision Transformers See Like Convolutional Neural Networks? (2021, positive): “Effective Receptive Fields: We conclude by computing effective receptive fields [23] for both ResNets and ViTs, with results in Figure 6 and Appendix C.”
On the Expressive Power of Overlapping Architectures of Deep Learning (2017, background): “Lastly, Luo et al. (2016) studied the effective total receptive field of different layers, a property of a similar nature to our total receptive field, where they measure the the degree to which each input pixel is affecting the output of each activation.”
Dynamic Filtering with Large Sampling Field for ConvNets (2018, background): “However, some applications such as large motion estimation and large object detection even require larger ERF.”