Relay Backpropagation for Effective Learning of Deep Convolutional Neural Networks

By Li Shen, Zhouchen Lin, Qingming Huang. European Conference on Computer Vision, 2015

336 citations recorded by the source corpus.

Learning deeper convolutional neural networks has become a tendency in recent years. However, many empirical evidences suggest that performance improvement cannot be attained by simply stacking more layers. In this paper, we consider the issue from an information theoretical perspective, and propose a novel method Relay Backpropagation, which encourages the propagation of effective information through the network in training stage. By virtue of the method, we achieved the first place in ILSVRC 2015 Scene Classification Challenge. Extensive experiments on two large scale challenging datasets demonstrate the effectiveness of our method is not restricted to a specific dataset or network architecture.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1f0bafe95728885034f5371420db2790e990971d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect (2020, neutral): “…encodes the natural inter-dependencies of classes — “TV” is indeed a good context for “controller” — any disrespect of it will hurt the feature representation learning [10], e.g. , re-weighting [13, 14] or re-sampling [15, 16] inevitably causes under-fitting to the head or over-fitting to the tail.”
Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss (2019, background): “, [46, 60, 5, 6] and references therein) and under-sampling the frequent classes (see, e.”
Decoupling Representation and Classifier for Long-Tailed Recognition (2019, background): “Various re-sampling strategies (Chawla et al., 2002; Shen et al., 2016; Cao et al., 2019), loss reweighting and margin regularization over few-shot classes are thus proposed.”
Squeeze-and-Excitation Networks (2017, methodology): “the effectiveness of SE blocks and follow the training and evaluation protocols described in [72], [74].”
Adaptive Risk Minimization: A Meta-Learning Approach for Tackling Group Shift (2020, background): “Instead, we draw inspiration from prior work in deep learning that demonstrates that uniformly sampling over a quantity of interest, such as labels or groups, is a strong method for achieving robustness and performance with respect to that quantity [60, 8, 55].”
Distribution Alignment: A Unified Framework for Long-tail Visual Recognition (2021, background): “Early works on re-balancing data distribution focus on learning one-stage models, which achieve limited successes due to lack of principled design in their strategies [2, 33, 3, 9, 26, 41].”
Deep Supervised Hashing for Fast Image Retrieval (2016, methodology): “To overcome this problem, we adopt class-aware sampling (Shen et al. 2016) in the training stage by randomly selecting 20 images from each category in each iteration.”
A systematic study of the class imbalance problem in convolutional neural networks (2017, background): “An oversampling approach specific to neural networks optimized with stochastic gradient descent is class-aware sampling [38].”
Imbalance Problems in Object Detection: A Review (2019, methodology): “[140] sample a mini-batch as uniform as possible from all classes also by restricting the same example and class to appear in the same order, which implies promoting the minority classes.”
Learning to Segment the Tail (2020, methodology): “[35] somewhat improve the performance for the tail classes.”
Learning to Model the Tail (2017, background): “A widespread yet suboptimal strategy is to resample and rebalance training data in the presence of the long tail, either by sampling examples from the rare classes more frequently [16, 17], or reducing the number of examples from the common classes [27].”
Simple Multi-dataset Detection (2021, background): “On the other hand, both OpenImages and Objects365 are longtailed and best train with advanced inter-dataset sampling strategy [26, 35], namely class-aware sampling.”
Exploring Balanced Feature Spaces for Representation Learning (2021, methodology): “…this performance bias, existing approaches either re-balance the data distribution through sampling (Chawla et al., 2002; Han et al., 2005; Shen et al., 2016; Mahajan et al., 2018) or the loss for each class (Cui et al., 2019; Khan et al., 2017; Cao et al., 2019; Khan et al., 2019) by…”
Temporal Segment Networks for Action Recognition in Videos (2017, neutral): “objects [7], [8], [9], scenes [10], [11], [12], and complex events [13], [14].”
Feature Space Augmentation for Long-Tailed Data (2020, background): “They typically have the advantage of a relatively clean implementation by either adjusting the loss function [9, 27] or manipulating the input batch [14, 37, 4, 50], and hence were widely adopted in practice [37, 27, 50].”
Convolutional Neural Networks: A Survey (2023, neutral): “During training, the CNN learns to associate the extracted features with the correct labels through a process of back propagation and optimization [35–38].”
Overcoming Classifier Imbalance for Long-Tail Object Detection With Balanced Group Softmax (2020, methodology): “One may consider using solutions to long-tail classification to overcome such an issue, including re-sampling training instances to balance the distribution [15, 7, 32, 26] and re-weighting classification loss at category level [5, 2, 18] or instance level [23, 33].”
Squeeze-and-Excitation Networks (2017, methodology): “In addition, we adopt the data balancing strategy described in [36] for mini-batch sampling.”
Contrastive Learning based Hybrid Networks for Long-Tailed Image Classification (2021, background): “Two types of re-sampling techniques are under-sampling [1, 28, 8] and over-sampling [1, 32, 31].”