Training Very Deep Networks

By Rupesh Kumar Srivastava, Klaus Greff, J. Urgen Schmidhuber. 2015

1,774 citations recorded by the source corpus.

Theoretical and empirical evidence indicates that the depth of neural networks is crucial for their success. However, training becomes more difficult as depth increases, and training of very deep networks remains an open problem. Here we introduce a new architecture designed to overcome this. Our so-called highway networks allow unimpeded information flow across many layers on information highways. They are inspired by Long Short-Term Memory recurrent networks and use adaptive gating units to regulate the information flow. Even with hundreds of layers, highway networks can be trained directly through simple gradient descent. This enables the study of extremely deep and efficient architectures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Deep Residual Learning for Image Recognition (2015, methodology): “Concurrent with our work, “highway networks” [42, 43] present shortcut connections with gating functions [15].”
Exploring the Limits of Language Modeling (2016, background): “…recently proposed sequence-to-sequence models employ conditional language models (Mikolov & Zweig, 2012) as their key component to solve diverse tasks like machine translation (Sutskever et al., 2014; Cho et al., 2014; Kalch-brenner et al., 2014) or video generation (Srivastava et al., 2015a).”
Dissecting Contextual Word Embeddings: Architecture and Representation (2018, background): “(2015), our character-toword encoder is a five-layer sub-module that first embeds single characters with an embedding layer then passes them through 2048 character n-gram CNN filters with max pooling, two highway layers (Srivastava et al., 2015), and a linear projection down to the model dimension.”
Deep Contextualized Word Representations (2018, methodology): “The context insensitive type representation uses 2048 character n-gram convolutional filters followed by two highway layers (Srivastava et al., 2015) and a linear projection down to a 512 representation.”
Adaptive Computation Time for Recurrent Neural Networks (2016, background): “make every network very deep and design its architecture in such a way as to mitigate the vanishing gradient problem [13] associated with long chains of iteration [29, 17].”
Generating Sentences from a Continuous Space (2015, methodology): “This technique is a variant of word dropout (Iyyer et al., 2015; Kumar et al., 2015), applied not to a feature extractor but to a decoder.”
HyperNetworks (2016, background): “ME n-gram (Mikolov et al., 2012) 1.37 Batch Norm LSTM (Cooijmans et al., 2016) 1.32 Recurrent Dropout LSTM (Semeniuta et al., 2016) 1.301 1.338 Zoneout RNN (Krueger et al., 2016) 1.27 HM-LSTM 3 (Chung et al., 2016) 1.27 (Sutskever et al., 2011) 1.60 GF-RNN (Chung et al., 2015) 1.58 20.0 M Grid-LSTM ”
Training Deep Nets with Sublinear Memory Cost (2016, background): “One common trend in those new models is to use deeper architectures [17, 13, 12, 9] to capture the complex patterns in a large amount of training data.”
Deep Networks with Stochastic Depth (2016, neutral): “vised [13] 7.97 - 1.92 33.70 Frac. Pool [25] - 27.62 - - All-CNN [6] 7.25 - - 41.20 Learning Activation [26] 7.51 30.83 - - R-CNN [27] 7.09 - 1.77 - Scalable BO [28] 6.37 27.40 1.77 - Highway Network [29] 7.60 32.24 - - Gen. Pool [30] 6.05 - 1.69 28.02 ResNet with constant depth 6.41 27.76 1.80 21.7”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, background): “(Lee et al., 2015), NiN (Lin et al., 2013), Maxout (Goodfellow et al., 2013), All-CNN (Springenberg et al., 2014), Highway Network (Srivastava et al., 2015) and Fractional Max-Pooling (Graham, 2014).”
Identity Mappings in Deep Residual Networks (2016, methodology): “We investigate the “exclusive” gates as used in [6,7] — the F path is scaled by g(x) and the shortcut path is scaled by 1−g(x).”
Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving (2019, background): “resent complex states in recurrent models (Hochreiter and Schmidhuber, 1997) and feed-forward models (Srivastava et al., 2015).”
Planning chemical syntheses with deep neural networks and symbolic AI (2017, methodology): “[35, 36] To assess its generalisation ability, we performed a time-split strategy.”
Densely Connected Convolutional Networks (2016, methodology): “We adopt a standard data augmentation scheme (mirroring/shifting) that is widely used for these two datasets [11, 13, 17, 22, 27, 20, 31, 33].”
Squeeze-and-Excitation Networks (2017, background): “Highway networks [40] employed a gating mechanism to regulate shortcut connections.”
Character-Aware Neural Language Models (2015, methodology): “Instead we obtained improvements by running y k through a highway network , recently proposed by Srivastava et al. (2015).”