Highway Networks

By Rupesh Kumar Srivastava, Klaus Greff, J. Urgen Schmidhuber. 2015

1,908 citations recorded by the source corpus.

There is plenty of theoretical and empirical evidence that depth of neural networks is a crucial ingredient for their success. However, network training becomes more difficult with increasing depth and training of very deep networks remains an open problem. In this extended abstract, we introduce a new architecture designed to ease gradient-based training of very deep networks. We refer to networks with this architecture as highway networks, since they allow unimpeded information flow across several layers on"information highways". The architecture is characterized by the use of gating units which learn to regulate the flow of information through a network. Highway networks with hundreds of layers can be trained directly using stochastic gradient descent and with a variety of activation functions, opening up the possibility of studying extremely deep and efficient architectures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Deep Residual Learning for Image Recognition (2015, background): “16) Highway [41, 42] 32 1.”
Massive Exploration of Neural Machine Translation Architectures (2017, methodology): “For deeper networks, we also experiment with two variants of residual connections (He et al., 2016; Srivastava et al., 2015) to encourage gradient flow.”
Neural Machine Translation in Linear Time (2016, background): “Stacked LSTM (Graves, 2013) 1.67 GF-LSTM (Chung et al., 2015) 1.58 Grid-LSTM (Kalchbrenner et al., 2016a) 1.47 Layer-normalized LSTM (Chung et al., 2016a) 1.46 MI-LSTM (Wu et al., 2016b) 1.44 Recurrent Memory Array Structures (Rocki, 2016) 1.40 HM-LSTM (Chung et al., 2016a) 1.40 Layer Norm HyperLSTM”
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, background): “Motivated by the idea of modeling diﬀerences between an intermediate layer’s output and the targets, which has shown to work well for many projects in the past [16, 21, 40], we introduce residual connections among the LSTM layers in a stack (see Figure 2).”
Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation (2016, background): “In computer vision, models with more than tens of convolution layers outperform the shallow ones on a series of image tasks in recent years (Srivastava et al., 2015; He et al., 2015; Szegedy et al., 2015).”
Neural GPUs Learn Algorithms (2015, methodology): “nit (GRU) (Cho et al., 2014). A GRU is similar to an LSTM, but its input and state are the same size, which makes it easier for us to generalize it later; a highway network could have also been used (Srivastava et al., 2015), but it lacks the reset gate. GRUs have shown performance similar to LSTMs ”
QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, methodology): “Following Seo et al. (2016), we also adopt a two-layer highway network (Srivastava et al., 2015) on top of this representation.”
Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, methodology): “All three use FastText (Joulin et al., 2016) pre-trained word embeddings 2 to which we apply a Highway transformation (Srivastava et al., 2015).”
Bidirectional Attention Flow for Machine Comprehension (2016, methodology): “The outputs of the Highway Network are two sequences of d - dimensional vectors, or more conveniently, two matrices: X ∈ R d × T for the context and Q ∈ R d × J for the query.”
Dual Co-Matching Network for Multi-choice Reading Comprehension (2019, methodology): “Gated mechanism (Srivastava, Greff, and Schmidhuber 2015) is used to fuse interaction representation into the original answer option representations.”
Wide Residual Networks (2016, methodology): “This problem was formulated as diminishing feature reuse in [26].”
Relation Networks for Object Detection (2017, methodology): “The backbone network [42, 45, 43, 25, 8, 53] is pre-trained on ImageNet classification task [11] and fine-tuned during detection training.”
Deep Networks with Stochastic Depth (2016, background): “Recently, several new architectures attempt to circumvent this problem through direct identity mappings between layers, which allow the network to pass on features unimpededly from earlier layers to later layers [8, 15].”
A-NICE-MC: Adversarial Training for MCMC (2017, background): “In particular, as shown in Figure 3, adding residual connections [20] and highway connections [21] to an existing model can significantly increase the time needed to transition between modes.”
Video Pixel Networks (2016, methodology): “Current approaches range from mean squared error models based on deep neural networks (Srivastava et al., 2015a; Oh et al., 2015), to models that predict quantized image patches (Ranzato et al., 2014), incorporate motion priors (Patraucean et al., 2015; Finn et al., 2016) or use adversarial losses…”
Identity Mappings in Deep Residual Networks (2016, methodology): “We investigate the “exclusive” gates as used in [6,7] — the F path is scaled by g(x) and the shortcut path is scaled by 1−g(x).”
The Reversible Residual Network: Backpropagation Without Storing Activations (2017, background): “Highway networks [29] circumvented this problem by introducing skip connections.”
Scattering Networks for Hybrid Representation Learning (2018, neutral): “Scat + WRN (ours) 93.1 Highway network [51] 92.4 All-CNN [50] 92.8 WRN 16 - 8 [59] 95.”
High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “Batch normalization downscales the residual branch: The combination of skip connections (Srivastava et al., 2015; He et al., 2016b;a) and batch normalization (Ioffe & Szegedy, 2015) enables us to train significantly deeper networks with thousands of layers (Zhang et al., 2019a).”