Learning to Skim Text

By Adams Wei Yu, Hongrae Lee, Quoc V. Le. 2017

128 citations recorded by the source corpus.

Recurrent Neural Networks are showing much promise in many sub-areas of natural language processing, ranging from document classification to machine translation to automatic question answering. Despite their promise, many recurrent models have to read the whole text word by word, making it slow to handle long documents. For example, it is difficult to use a recurrent network to read a book and answer questions about it. In this paper, we present an approach of reading text while skipping irrelevant information if needed. The underlying model is a recurrent network that learns how far to jump after reading a few words of the input text. We employ a standard policy gradient method to train the model to make discrete jumping decisions. In our benchmarks on four different tasks, including number prediction, sentiment analysis, news article classification and automatic Q&A, our proposed model, a modified LSTM with jumping, is up to 6 times faster than the standard sequential LSTM, while maintaining the same or even better accuracy.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, methodology): “nce paper at ICLR 2018 2014) or Long Short Term Memory architectures (Hochreiter & Schmidhuber, 1997). For simple tasks such as text classiﬁcation, with reinforcement learning techniques, models (Yu et al., 2017) have been proposed to skip irrelevant tokens to both further address the long depen”
Dynamic Neural Networks: A Survey (2021, methodology): “Specifically, LSTM-Jump [174] implements an auxiliary unit to predict the jumping stride within a defined range, and the reading process ends when the unit outputs zero.”
Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision (2016, background): “reinforcement learning (Sutton and Barto, 1998) in order to directly optimize the task reward of the structured prediction problem (Norouzi et al., 2016; Li et al., 2016; Yu et al., 2017).”
Dilated Recurrent Neural Networks (2017, background): “Recently, Yu et al. [27] proposed learning based RNNs with the ability to jump (skim input text) after seeing a few timestamps worth of data; although the authors have shown that the modiﬁed LSTM with jumping provides up to a six-fold speed increase, the efﬁciency gain is mainly in the testing…”
Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (2019, methodology): “The idea of multi-step reasoning has been explored in many tasks, including image classiﬁ-cation [34], text classiﬁcation [54], image generation [17], language-based image editing [6], Visual Question Answering (VQA) [53, 36], and Machine Reading Comprehension (MRC) [8, 12, 20, 44, 41, 31].”
PonderNet: Learning to Ponder (2021, background): “A parallel line of research has explored using similar techniques to reduce the computation by skipping elements from a sequence of processed inputs (Yu et al., 2017; Campos Camunez et al., 2018).”
Look Harder: A Neural Machine Translation Model with Hard Attention (2019, background): “Even though RL based models are difﬁcult to train, in recent years, multiple works (Mnih et al., 2014; Choi et al., 2017; Yu et al., 2017; Narayan et al., 2018; Sathish et al., 2018; Shen et al., 2018) have shown to improve the performance of several nat-ural language processing tasks.”
Skipping RNN State Updates without Retraining the Original Model (2019, methodology): “Similar to prior work in the domain of skipping RNN time-steps [1, 4, 11, 17, 18], we use the hidden state from the previous time-step and the input consumed during the current time-step as the input to the predictor.”
Evidence Extraction for Machine Reading Comprehension with Deep Probabilistic Logic (2018, result): “Yu et al. (2017) and Seo et al. (2017) learn to skip some tokens in neural models, which is similar with ours in the sense that we drop some useless sentences, but the difference is that the skipping module is trained jointly with the downstream task and the dropping unit is token, while in our…”
Neural Speed Reading with Structural-Jump-LSTM (2019, unknown): “This combined approach of both skipping and jumping according to text structure yields notable gains in efﬁciency (reduced FLOPs) without loss of effectiveness (accuracy).”
Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks (2017, unknown): “Unlike LSTM-Jump, our proposed approach is differentiable, thus not requiring any modifications to the loss function and simplifying the optimization process, and is not limited to a predefined set of sample selection patterns.”
Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation (2021, methodology): “…Neural Machine Translation (NMT), Reinforcement Learning (RL) has gained attraction due to the suitability of Policy Gradient (PG) methods for the end-to-end training paradigm (Ranzato et al., 2016; Li et al., 2016; Yu et al., 2017; Li et al., 2018; Flachs et al., 2019; Sankar and Ravi, 2019).”
A Survey on Dynamic Neural Networks for Natural Language Processing (2022, background): “Operation options LSTM-Jump (Yu et al., 2017) hidden states skip multiple steps; stop reading Skip RNN (Campos et al., 2018) states of the update gate; hidden states skip a single step ReasoNet (Shen et al., 2017) hidden states stop reading Jumper (Liu et al., 2018) input sentence; hidden states sto”
Evidence Sentence Extraction for Machine Reading Comprehension (2019, background): “Several work also investigate content selection at the token level (Yu et al., 2017; Seo et al., 2018), in which some tokens are automatically skipped by neural models.”
Neural Speed Reading via Skim-RNN (2017, methodology): “LSTM-Jump (Yu et al., 2017) has the same goal as our model in that it aims to reduce the computational cost of recurrent neural networks.”
Speed Reading: Learning to Read ForBackward via Shuttle (2018, methodology): “For a fair comparison with (Yu et al., 2017), the dropout rate between LSTM layers was set to 0.”
Surprisal-Triggered Conditional Computation with Neural Networks (2020, background): “This often takes the form of a mixture-of-experts [23, 24], in which a learned controller is used to select the experts relevant for a given input [25, 26, 27, 28, 29, 30], possibly hierarchically [31, 32, 33, 34].”
Interactive Machine Comprehension with Information Seeking Agents (2019, background): “Skip-reading (Yu et al., 2017; Seo et al., 2017; Choi et al., 2017) is an existing setting in which MRC models read partial documents.”