Depth-Gated LSTM

By Kaisheng Yao, Trevor Cohn, Ekaterina Vylomova, Kevin Duh, Chris Dyer. 2015

78 citations recorded by the source corpus.

In this short note, we present an extension of long short-term memory (LSTM) neural networks to using a depth gate to connect memory cells of adjacent layers. Doing so introduces a linear dependence between lower and upper layer recurrent units. Importantly, the linear dependence is gated through a gating function, which we call depth gate. This gate is a function of the lower layer memory cell, the input to and the past memory cell of this layer. We conducted experiments and verified that this new architecture of LSTMs was able to improve machine translation and language modeling performances.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation (2016, background): “Yao et al. (2015) introduced depth-gated shortcuts connecting LSTM cells at adjacent layers to provide a fast way to propagate the gradients.”
Long Short-Term Memory-Networks for Machine Reading (2016, methodology): “Finally, we implemented more sophisticated LSTM architectures, such as a gated-feedback LSTM (gLSTM; Chung et al. (2015)) and a depth-gated LSTM (dLSTM; Yao et al. (2015)).”
Attention with Intention for a Neural Network Conversation Model (2015, background): “All of the recurrent networks are implemented using a recently proposed depth-gated long-shortterm memory (LSTM) network [16].”
Highway long short-term memory RNNS for distant speech recognition (2015, background): “After developing the highway LSTMs independently we noticed that similar work has been done in [16, 17, 18].”
Shortcut Sequence Tagging (2017, background): “In this section, we ﬁrst introduce the traditional stacked LSTMs. Based on this architecture, we propose a shortcut block structure, which is the basic element of our stacked models.”
Incorporating Side Information into Recurrent Neural Network Language Models (2016, background): “…such as: Long Short Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997), Gated Recurrent Unit (GRU) (Cho et al., 2014), or recently deeper structures, e.g. Depth Gated Long Short Term Memory (DGLSTM) – a stack of LSTMs with extra connections between memory cells in deep layers (Yao et al., 2015).”
Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions (2019, methodology): “Apart from them, other variants have been introduced such as Depth Gated LSTM, Clockwork RNN, Stochastic Recurrent Networks and Bidirectional RNN (Yao et al., 2015; Koutńık et al., 2014; Bayer and Osendorfer, 2014; Schuster and Paliwal, 1997).”
Topically Driven Neural Language Model (2017, unknown): “(8)Multi-layer LSTMs are vanilla stacked LSTMs without skip connections (Gers and Schmidhuber, 2000) or depthgating (Yao et al., 2015).”
Recent Advances in Recurrent Neural Networks (2017, background): “In stacked LSTM, a stack pointer can determine which cell in the LSTM provides state and memory cell of a previous timestep [58].”