Fast Parametric Learning with Activation Memorization
By Jack W. Rae, Chris Dyer, Peter S. Dayan, Timothy P. Lillicrap. 2018
47 citations recorded by the source corpus.
Neural networks trained with backpropagation often struggle to identify classes that have been observed a small number of times. In applications where most class labels are rare, such as language modelling, this can become a performance bottleneck. One potential remedy is to augment the network with a fast-learning non-parametric model which stores recent activations and class labels into an external memory. We explore a simplified architecture where we treat a subset of the model parameters as fast memory stores. This can help retain information over longer time intervals than a traditional memory, and does not require additional space or compute. In the case of image classification, we display faster binding of novel classes on an Omniglot image curriculum task. We also show improved performance for word-based language models on news reports (GigaWord), books (Project Gutenberg) and Wikipedia articles (WikiText-103) --- the latter achieving a state-of-the-art perplexity of 29.2.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- J. Urgen Schmidhuber
- Ruslan Salakhutdinov
- Christopher R'e
- Quoc V. Le
- Pieter Abbeel
- Razvan Pascanu
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Andrew Zisserman
- Xiaogang Wang
- Jitendra Malik
- Ming-Hsuan Yang
- Li Fei-Fei
In Optimization & Training
- Razvan Pascanu
- Christopher R'e
- Timothy P. Lillicrap
- Oriol Vinyals
- Yoshua Bengio
- Anna Potapenko
- Ali Razavi
- David Raposo
Citations in the corpus
- Compressive Transformers for Long-Range Sequence Modelling (2019, background): “This dataset has been a popular recent LM benchmark due to the potential to exploit longer-range dependencies (Grave et al., 2016; Rae et al., 2018; Bai et al., 2018b).”
- Relational recurrent neural networks (2018, methodology): “For WikiText-103, we see this can be compared to LSTM architectures [5, 29], convolutional models [39] and hybrid recurrent-convolutional models [40].”
- Augmenting Self-attention with Persistent Memory (2019, neutral): “[33] – LSTM + Hebbian + Cache - 29.”
- Efficiently Modeling Long Sequences with Structured State Spaces (2021, methodology): “The RNN baselines included in Table 8 are the AWD-QRNN [27], an efficient linear gated RNN, and the LSTM + Cache + Hebbian + MbPA [33], the best performing pure RNN in the literature.”
- Dynamic Evaluation of Transformer Language Models (2019, neutral): “n-parametric 4 Model # of params valid test LSTM+ neural cache (Grave et al., 2017b) - - 40.8 GCNN-14 (Dauphin et al., 2017) - - 37.2 QRNN (Merity et al., 2018) 151M 32.0 33.0 LSTM + hebbian + cache (Rae et al., 2018) - 29.7 29.9 Transformer + adaptive input (Baevski and Auli, 2019) 247M 19.8 20.5 T”
- Do Transformers Need Deep Long-Range Memory? (2020, background): “E.g. a typical state-of-the-art LSTM language model state size may range from 4KB (Rae et al., 2018) to model Wikipedia articles to 64KB (Jozefowicz et al., 2016) to model news — and is never greater than 1MB.”
- Segatron: Segment-Aware Transformer for Language Modeling and Understanding (2020, unknown): “Hebbian+Cache (Rae et al. 2018) - 29.”
- Limits to Depth Efficiencies of Self-Attention (2020, unknown): “2 [Rae et al., 2018].”