Deeply-Supervised Nets

By Chen-Yu Lee, Saining Xie, Patrick W. Gallagher, Zhengyou Zhang, Zhuowen Tu. 2014

2,494 citations recorded by the source corpus.

Our proposed deeply-supervised nets (DSN) method simultaneously minimizes classification error while making the learning process of hidden layers direct and transparent. We make an attempt to boost the classification performance by studying a new formulation in deep networks. Three aspects in convolutional neural networks (CNN) style architectures are being looked at: (1) transparency of the intermediate layers to the overall classification; (2) discriminativeness and robustness of learned features, especially in the early layers; (3) effectiveness in training due to the presence of the exploding and vanishing gradients. We introduce "companion objective" to the individual hidden layers, in addition to the overall objective at the output layer (a different strategy to layer-wise pre-training). We extend techniques from stochastic gradient methods to analyze our algorithm. The advantage of our method is evident and our experimental result on benchmark datasets shows significant performance gain over existing methods (e.g. all state-of-the-art results on MNIST, CIFAR-10, CIFAR-100, and SVHN).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Rethinking the Inception Architecture for Computer Vision (2015, background): “Also Lee et al[11] argues that auxiliary classiﬁers promote more stable learning and better convergence.”
Training Very Deep Networks (2015, background): “This enables the study of extremely deep and efficient architectures.”
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015, background): “In [29, 18], auxiliary classiﬁers are added to intermediate layers to help with convergence.”
FitNets: Hints for Thin Deep Nets (2014, methodology): “More recently, (Chen-Yu et al., 2014; Szegedy et al., 2014; Gulcehre & Bengio, 2013) showed that adding supervision to intermediate layers of deep architectures assists the training of deep networks.”
Rethinking ImageNet Pre-Training (2018, background): “They designed a specialized detector driven by deeply supervised networks [24] and dense connections [18].”
Wide Residual Networks (2016, background): “We call the resulting network structures wide residual networks (WRNs) and show that these are far superior over their commonly used thin and very deep counterparts.”
HyperNetworks (2016, background): “Network in Network (Lin et al., 2014) 8.81% FitNet (Romero et al., 2014) 8.39% Deeply Supervised Nets (Lee et al., 2015) 8.22% Highway Networks (Srivastava et al., 2015) 7.72% ELU (Clevert et al., 2015) 6.55% Original Resnet-110 (He et al., 2016a) 6.43% 1.7 M Stochastic Depth Resnet-110 (Huang et…”
Deep Networks with Stochastic Depth (2016, methodology): “Several approaches exist to reduce this eﬀect in practice, for example through careful initialization [12], hidden layer supervision [13], or, recently, Batch Normalization [14].”
Colorful Image Colorization (2016, methodology): “We use deep supervision to help guide learning [26].”
Density estimation using Real NVP (2016, methodology): “This follows a philosophy similar to guiding intermediate layers using intermediate classiﬁers [33], and having multiple layers of latent variables which represent different levels of abstraction [46, 42].”
Dual Contradistinctive Generative Autoencoder (2020, positive): “Inspired by [44], we utilize information from feature maps at different scales.”
The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation (2016, background): “…DenseNets perform deep supervision thanks to short paths to all feature maps in the architecture (similar to Deeply Supervised Networks [17]); and (3) feature reuse, all layers can easily access their preceding layers making it easy to reuse the information from previously computed…”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, background): “(Lee et al., 2015), NiN (Lin et al., 2013), Maxout (Goodfellow et al., 2013), All-CNN (Springenberg et al., 2014), Highway Network (Srivastava et al., 2015) and Fractional Max-Pooling (Graham, 2014).”
Identity Mappings in Deep Residual Networks (2016, neutral): “81 DSN [16] 8.”
DensePose: Dense Human Pose Estimation in the Wild (2018, background): “Cascading can improve performance both by providing context to the following stages, and also through the beneﬁts of deep supervision [24].”