Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems

By Colin Raffel, D. Ellis. arXiv.org, 2015

322 citations recorded by the source corpus.

We propose a simplified model of attention which is applicable to feed-forward neural networks and demonstrate that the resulting model can solve the synthetic "addition" and "multiplication" long-term memory problems for sequence lengths which are both longer and more widely varying than the best published results for these tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/87119572d1065fb079e1dee8fcdb6c4811143f96

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Monotonic Chunkwise Attention (2017, unknown): “This use of the encoder’s hidden states as a memory gives the model the ability to bridge long input-output time lags (Raffel & Ellis, 2015), which provides a distinct advantage over sequence-to-sequence models lacking an attention mechanism (Bahdanau et al., 2015).”
Text-based adventures of the golovin AI agent (2017, methodology): “We use the LSTM neural networks operating on words [25], augmented by the attention mechanism ([26] and [27]).”
Attention-based Deep Multiple Instance Learning (2018, unknown): “Interestingly, the proposed MIL pooling corresponds to a version of the attention mechanism (Lin et al., 2017; Raffel & Ellis, 2015).”
Pythia: AI-assisted Code Completion System (2019, methodology): “. 6.3 Tuning neural network architecture Architecture of the neural network is a hyperparameter. Besides LSTM, we consider gated recurrent units (GRU), LSTM with attention mechanism for temporal data [20], and variations of classiﬁcation layer following the LSTM. Selecting the best model for serving”
DataStories at SemEval-2017 Task 4: Deep LSTM with Attention for Message-level and Topic-based Sentiment Analysis (2017, methodology): “In order to amplify the contribution of important elements in the ﬁnal representation we use an attention mechanism (Rock-täschel et al., 2015; Raffel and Ellis, 2015), that aggregates all the hidden states using their relative importance (see Figure 2).”
Deep Speaker Embeddings for Short-Duration Speaker Verification (2017, methodology): “In this work we propose to aggregate the hidden activations of the network using a feedforward attention model [16].”
Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks (2020, background): “Different from traditional attention mechanism that all instances are sequentially dependent [44, 45],”
A Neural-Network-Based Method for RUL Prediction and SOH Monitoring of Lithium-Ion Battery (2019, background): “By introducing a self-attention mechanism [33], the distraction problem can be effectively reduced, and the accuracy of the model will improve.”
Deep Learning with Dynamic Computation Graphs (2017, background): “Recently, Raffel & Ellis (2016) have introduced an attention model for feed-forward neural networks.”