Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks

By Víctor Campos, Brendan Jou, Xavier Giró-i-Nieto, Jordi Torres, Shih-Fu Chang. 2017

236 citations recorded by the source corpus.

Recurrent Neural Networks (RNNs) continue to show outstanding performance in sequence modeling tasks. However, training RNNs on long sequences often face challenges like slow inference, vanishing gradients and difficulty in capturing long term dependencies. In backpropagation through time settings, these issues are tightly coupled with the large, sequential computational graph resulting from unfolding the RNN in time. We introduce the Skip RNN model which extends existing RNN models by learning to skip state updates and shortens the effective size of the computational graph. This model can also be encouraged to perform fewer state updates through a budget constraint. We evaluate the proposed model on various tasks and show how it can reduce the number of required RNN updates while preserving, and sometimes even improving, the performance of the baseline RNN models. Source code is publicly available at this https URL .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, methodology): “allocating cheap operations to unimportant frames) is similar to the aforementioned text skimming [167], [168].”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, background): “…techniques for recurrent networks have been introduced and continue to be actively explored (El Hihi & Bengio, 1995; Schuster & Paliwal, 1997; Gers et al., 2002; Koutnik et al., 2014; Le et al., 2015; Ba et al., 2016; Wu et al., 2016; Krueger et al., 2017; Merity et al., 2017; Campos et al., 2018).”
PonderNet: Learning to Ponder (2021, background): “A parallel line of research has explored using similar techniques to reduce the computation by skipping elements from a sequence of processed inputs (Yu et al., 2017; Campos Camunez et al., 2018).”
Focused Hierarchical RNNs for Conditional Sequence Processing (2018, background): “The Skip-RNN (Campos et al., 2017) learns an updating rate by predicting how many steps to skip in the future.”
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (2023, methodology): “For recurrent neural networks, one method is to skip multiple tokens at a given time step (Campos et al., 2017; Seo et al., 2017; Hansen et al., 2019).”
Conditional-Computation-Based Recurrent Neural Networks for Computationally Efficient Acoustic Modelling (2018, methodology): “We experimented with 2 previous work models, Hierarchical-Multiscale LSTM and Skip-GRU, and a pro- Figure 2: Internal states of our proposed model (forward network) versus time for a portion of a TIMIT test utterance.”
Speed Reading: Learning to Read ForBackward via Shuttle (2018, background): “(Campos et al., 2018) also attempt to dynamically control the RNN’s computational costs, but they instead control the number of units to update at each time step.”
Surprisal-Triggered Conditional Computation with Neural Networks (2020, methodology): “To avoid the difﬁculties of reinforcement learning, often a soft approximation to the hard selection decision is used during training [11, 35, 36, 37, 38, 39, 40, 41, 42].”
Skipping RNN State Updates without Retraining the Original Model (2019, methodology): “Similar to prior work in the domain of skipping RNN time-steps [1, 4, 11, 17, 18], we use the hidden state from the previous time-step and the input consumed during the current time-step as the input to the predictor.”
FrameExit: Conditional Early Exiting for Efficient Video Recognition (2021, background): “This is commonly done by training agents to find which frame to observe next [55, 52, 57, 8], ranking frames based on saliency [29], gating frames [25], skipping RNN states [4], or by using audio to select relevant frames [15, 29].”
Time2Vec: Learning a Vector Representation of Time (2019, background): “2) Event-MNIST: Sequential (event-based) MNIST is a common benchmark in sequence modeling literature (see, e.g. , [4, 6, 15]).”
Learning Longer-term Dependencies in RNNs with Auxiliary Losses (2018, background): “A more recent approach is to skip input information at certain steps (Yu et al., 2017; Seo et al., 2018; Campos et al., 2018).”
Variable-rate discrete representation learning (2021, background): “Other models operate on ﬁxed-quantum inputs, but adaptively allocate capacity 5 (Eslami et al., 2016; Graves, 2016; Yeung et al., 2016; Campos et al., 2017; Fig-urnov et al., 2017; Malinowski et al., 2018; Marin et al., 2019; Kirillov et al., 2020; Lugosch et al., 2020).”
Neural Speed Reading via Skim-RNN (2017, result): “We also note the existence of Skip-LSTM (Campos et al., 2017), a recent, concurrent submission to ours that shares many characteristics with LSTM-Jump.”
Neural Speed Reading with Structural-Jump-LSTM (2019, unknown): “…Campos et al., 2018) where the model learns what parts of the image it should focus on, to natural language processing (NLP), such as text classiﬁcation and question answering (Yu et al., 2017; Campos et al., 2018; Yu et al., 2018), where the model learns which part of a document it can ignore.”
Sparsely Changing Latent States for Prediction and Planning in Partially Observable Domains (2021, methodology): “Binarized update gates: Closely related to our ReTanh, Skip RNNs [35] use a binary gate to determine latent state update decisions.”