Associative Long Short-Term Memory

By Ivo Danihelka, Greg Wayne, Benigno Uria, Nal Kalchbrenner, Alex Graves. 2016

196 citations recorded by the source corpus.

We investigate a new method to augment recurrent neural networks with extra memory without increasing the number of network parameters. The system has an associative memory based on complex-valued vectors and is closely related to Holographic Reduced Representations and Long Short-Term Memory networks. Holographic Reduced Representations have limited capacity: as they store more information, each retrieval becomes noisier due to interference. Our system in contrast creates redundant copies of stored information, which enables retrieval with reduced noise. Experiments demonstrate faster learning on multiple memorization tasks.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

Using Fast Weights to Attend to the Recent Past (2016, background): “Danihelka et al. [2016] extended LSTMs by adding complex associative memory.”
Full Resolution Image Compression with Recurrent Neural Networks (2016, background): “Associative LSTM: Another neural network element we examine is the Associative LSTM [5].”
Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks (2019, background): “…real-valued representations (Vaswani et al., 2017; Bahdanau et al., 2014; Parikh et al., 2016), there have also been emerging interest in complex (Danihelka et al., 2016; Arjovsky et al., 2016; Gaudet and Maida, 2017) and hypercom-plex representations (Parcollet et al., 2018b,a; Gaudet and…”
Neural Program Meta-Induction (2017, methodology): “…four techniques Novel Architectures Applied to Program Induction NTM [7], Stack RNN [11], NRAM [12] Neural Transducers [8], Learn Algo [21] Others [4, 20, 2, 13] Trace-Augmented Induction NPI [18] Recursive NPI [3], NPL [14] Non Example-Driven Induction (e.g., Natural Language-Driven Induction)…”
Toward an Integration of Deep Learning and Neuroscience (2016, background): “…architectures 53 , which were proposed in cognitive science (Plate, 1995; Stewart and Eliasmith, 2009; Eliasmith, 2013), are also being considered in the context of eﬃciently-trainable artiﬁcial neural networks (Danihelka et al., 2016)—in eﬀect, these systems learn how to use variable binding.”
Deep Complex Networks (2017, background): “…due to their potential to enable easier optimization [Nitta, 2002], better generalization characteristics [Hirose and Yoshida, 2012], faster learning [Arjovsky et al., 2015, Danihelka et al., 2016, Wisdom et al., 2016] and to allow for noise-robust memory mechanisms [Danihelka et al., 2016].”
Lie Access Neural Turing Machine (2016, background): “[4] designed an external memory based on holographic reduced representations, which can store unlimited memory but the larger the size the more noisy the retrieval.”
A survey on long short-term memory networks for time series prediction (2021, background): “An LSTM concept with external memory is presented by [26] that augments the network without an increase in parameters.”
A comprehensive study of deep bidirectional LSTM RNNS for acoustic modeling in speech recognition (2016, background): “Results in Table 7 show the improvement by LSTMs and also with an associative LSTM [47].”
Quaternion Recurrent Neural Networks (2018, methodology): “More recently, neural networks of complex and hypercomplex numbers have received an increasing attention with motivations described in [7, 13, 35, 36], and some efforts have shown promising results in different applications.”
Revisiting NARX Recurrent Neural Networks for Long-Term Dependencies (2017, methodology): “, 2014), using associative or explicit memory (Plate, 1993; Danihelka et al., 2016; Graves et al., 2014; Weston et al., 2015; Sukhbaatar et al., 2015), and initializing or restricting weight matrices to be orthogonal (Arjovsky et al.”
Analyzing and Exploiting NARX Recurrent Neural Networks for Long-Term Dependencies (2017, methodology): “…datasets in the domains of surgical activity recognition and phoneme recognition, and then we consider the copy problem and the sequential permuted MNIST task, as they have been used extensively to test RNN architectures for their ability to capture long-term dependencies [21, 29, 24, 1, 19, 7].”
An optical neural chip for implementing complex-valued neural network (2021, background): “Meanwhile, recent studies suggest that complex-valued arithmetic 7,8 would signi ﬁ cantly improve the performance of neural networks by offering rich representational capacity 9 , fast convergence 10 , strong generalization 11 and noise-robust memory mechanisms 12 .”
Deep Reinforcement Learning: An Overview (2017, background): “…and an idea similar to target network, ”soft” target, which, rather than copying the weights directly as in DQN, updates the soft target network weights θ ′ slowly to track the learned networks weights θ : The authors adapted batch normalization to handle the issue that the different components…”
Knowledge Graph Completion via Complex Tensor Factorization (2017, background): “In deep learning, Danihelka et al. (2016) proposed an LSTM extended with an associative memory based on complex-valued vectors for memorization tasks, and Hu et al. (2016) a complex-valued neural network for speech synthesis.”
Returnn: The RWTH extensible training framework for universal recurrent neural networks (2016, background): “Different cell implementations including (one-and two-dimensional) LSTM, gated recurrent units (GRU) [25], associative LSTM [26] and many more variants are available.”
Hopfield Networks is All You Need (2020, methodology): “The Hopfield layer in a deep neural network architecture can implement: • a memory (storage) with associative retrieval [20, 5], • conditional pooling and averaging operations [67, 39], • combining data by associations [2], • associative credit assignment (e.”