Neural Speed Reading via Skim-RNN

By Minjoon Seo, Sewon Min, Ali Farhadi, Hannaneh Hajishirzi. 2017

81 citations recorded by the source corpus.

Inspired by the principles of speed reading, we introduce Skim-RNN, a recurrent neural network (RNN) that dynamically decides to update only a small fraction of the hidden state for relatively unimportant input tokens. Skim-RNN gives computational advantage over an RNN that always updates the entire hidden state. Skim-RNN uses the same input and output interfaces as a standard RNN and can be easily used instead of RNNs in existing models. In our experiments, we show that Skim-RNN can achieve significantly reduced computational cost without losing accuracy compared to standard RNNs across five different natural language tasks. In addition, we demonstrate that the trade-off between accuracy and speed of Skim-RNN can be dynamically controlled during inference time in a stable manner. Our analysis also shows that Skim-RNN running on a single CPU offers lower latency compared to standard RNNs on GPUs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Natural Language Processing

In Model Architectures

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, methodology): “Language Sa & Te Language modeling [11], [16], [118], [170], [172], Machine translation [16], [35], [36], Classification [64], [65], [174], Processing Sentiment analysis [168], [169], [171], [175], [176], Question answering [35], [63], [168], [171], [173]”
The Right Tool for the Job: Matching Model and Instance Complexities (2020, methodology): “Some works introduced methods to allocate fewer resources to certain parts of the input (e.g., certain words), thereby potentially reducing training and/or inference time (Graves, 2016; Seo et al., 2018).”
How to Fine-Tune BERT for Text Classification? (2019, background): “We compare our model with the following a variety of different methods: CNN-based methods such as Char-level CNN (Zhang et al., 2015), VDCNN (Conneau et al., 2016) and DPCNN (Johnson and Zhang, 2017); RNN-based models such as D-LSTM (Yogatama et al., 2017), Skim-LSTM (Seo et al., 2017) and hierarchi”
Measuring and Narrowing the Compositionality Gap in Language Models (2022, methodology): “Allowing models to use more computation to solve harder problems has been previously explored in both language (Graves, 2016; Seo et al., 2017; Dong et al., 2019; Schwartz et al., 2020; Schwarzschild et al., 2021; Nye et al., 2021; Wei et al., 2022b; Zhou et al., 2022) and vision (Bolukbasi et al.,…”
Speed Reading: Learning to Read ForBackward via Shuttle (2018, background): “As in our work, instead of skimming some words, (Seo et al., 2018) consider all words but use a small RNN for irrelevant words and the original large RNN for relevant ones.”
Learning Longer-term Dependencies in RNNs with Auxiliary Losses (2018, background): “A more recent approach is to skip input information at certain steps (Yu et al., 2017; Seo et al., 2018; Campos et al., 2018).”
A Survey on Dynamic Neural Networks for Natural Language Processing (2022, methodology): “;) or (.!?) PoWER (Goyal et al., 2020) attention drop tokens TR-BERT (Ye et al., 2021) hidden states forward tokens LAT (Kim and Cho, 2021) attention forward tokens LTP (Kim et al., 2022) attention drop tokens Transkimmer (Guan et al., 2022) hidden states forward tokens VCRNN (Jernite et al., 2017) ”
Interactive Machine Comprehension with Information Seeking Agents (2019, background): “Skip-reading (Yu, Lee, and Le 2017; Seo et al. 2017; Choi et al. 2017) is an existing setting in which MRC models read partial documents.”
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (2023, methodology): “For recurrent neural networks, one method is to skip multiple tokens at a given time step (Campos et al., 2017; Seo et al., 2017; Hansen et al., 2019).”
Skipping RNN State Updates without Retraining the Original Model (2019, methodology): “There have been multiple works on skipping RNN state updates [1, 4, 11, 17, 18].”
Large-Scale Answerer in Questioner's Mind for Visual Dialog Question Generation (2019, methodology): “In specific, we can apply diverse methods such as skipping the update of hidden states in some steps (Seo et al., 2018), using convolution networks or self-attention networks (Yu et al.”
Neural Speed Reading with Structural-Jump-LSTM (2019, unknown): “We calculate the total FLOPs used by the models as done by Seo et al. (2018) and Yu et al. (2018), reported as a FLOP reduction (FLOP-r) between the full read and speed read model.”
Evidence Sentence Extraction for Machine Reading Comprehension (2019, background): “Several work also investigate content selection at the token level (Yu et al.; Seo et al., 2018), in which some tokens are automatically skipped by neural models.”
Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention (2020, methodology): “Further approaches include sentence representation learning using surprisal and part of speech tags as proxies to human attention [76], attention as a way to improve time complexity for NLP tasks [68], and learning saliency scores by training for sentence comparison [66].”
Description Based Text Classification with Reinforcement Learning (2020, unknown): “4 – Skim-LSTM (Seo et al., 2018) 6.”