Neural Speed Reading with Structural-Jump-LSTM

By Christian Hansen, Casper Hansen, Stephen Alstrup, Jakob Grue Simonsen, Christina Lioma. 2019

41 citations recorded by the source corpus.

Recurrent neural networks (RNNs) can model natural language by sequentially 'reading' input tokens and outputting a distributed representation of each token. Due to the sequential nature of RNNs, inference time is linearly dependent on the input length, and all inputs are read regardless of their importance. Efforts to speed up this inference, known as 'neural speed reading', either ignore or skim over part of the input. We present Structural-Jump-LSTM: the first neural speed reading model to both skip and jump text during inference. The model consists of a standard LSTM and two agents: one capable of skipping single words when reading, and one capable of exploiting punctuation structure (sub-sentence separators (,:), sentence end symbols (.!?), or end of text markers) to jump ahead after reading a word. A comprehensive experimental evaluation of our model against all five state-of-the-art neural reading models shows that Structural-Jump-LSTM achieves the best overall floating point operations (FLOP) reduction (hence is faster), while keeping the same accuracy or even improving it compared to a vanilla LSTM that reads the whole text.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Shiji Song
2. Yulin Wang
3. Gao Huang
4. Julian McAuley
5. Le Yang
6. Yizeng Han
7. Canwen Xu
8. Honghui Wang

In Natural Language Processing

1. Gao Huang
2. Shiji Song
3. Julian McAuley
4. Yulin Wang
5. Canwen Xu
6. Yongfeng Zhang
7. Xu Chen
8. Yann LeCun

In Model Architectures

1. Shiji Song
2. Yulin Wang
3. Gao Huang
4. Honghui Wang
5. Le Yang
6. Julian McAuley
7. Yizeng Han
8. Canwen Xu

Citations in the corpus

• Dynamic Neural Networks: A Survey (2021, methodology): “Without training the RNNs and the controllers jointly as in [167] and [168], a predictor is trained in [169] to estimate”
• Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (2023, methodology): “For recurrent neural networks, one method is to skip multiple tokens at a given time step (Campos et al., 2017; Seo et al., 2017; Hansen et al., 2019).”
• A Survey on Dynamic Neural Networks for Natural Language Processing (2022, background): “Jump-LSTM (Hansen et al., 2019) hidden states stop reading; jump to next (,”