Hierarchical Memory Networks

By A. P. Sarath Chandar, Sungjin Ahn, Hugo Larochelle, Pascal Vincent, G. Tesauro, Yoshua Bengio. 2016

88 citations recorded by the source corpus.

Memory networks are neural networks with an explicit memory component that can be both read and written to by the network. The memory is often addressed in a soft way using a softmax function, making end-to-end training with backpropagation possible. However, this is not computationally scalable for applications which require the network to read from extremely large memories. On the other hand, it is well known that hard attention mechanisms based on reinforcement learning are challenging to train successfully. In this paper, we explore a form of hierarchical memory network, which can be considered as a hybrid between hard and soft attention memory networks. The memory is organized in a hierarchical structure such that reading from it is done with less computation than soft attention over a flat memory, while also being easier to train than hard attention over a flat memory. Specifically, we propose to incorporate Maximum Inner Product Search (MIPS) in the training and inference procedures for our hierarchical memory network. We explore the use of various state-of-the art approximate MIPS techniques and report results on SimpleQuestions, a challenging large scale factoid question answering task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Model Architectures

Citations in the corpus

Reformer: The Efficient Transformer (2020, methodology): “The original implementation of memory networks (Weston et al., 2014) and later work on scaling it (Bordes et al., 2015; Chandar et al., 2016) used memory with size in the millions.”
Learning to Remember Rare Events (2017, methodology): “The original implementation of memory networks (Weston et al., 2014) and later work on scaling it (Bordes et al., 2015; Chandar et al., 2016) used memory with size in the millions.”
Tracking the World State with Recurrent Entity Networks (2016, methodology): “Our model is similar to a Memory Network and its variants (Weston et al., 2014; Sukhbaatar et al., 2015; Chandar et al., 2016; Miller et al., 2016) in the way it produces an output using a softmax over blocks of hidden states, and our encoding layer is inspired by techniques used in those works.”
Augmenting Transformers with KNN-Based Composite Memory for Dialog (2020, neutral): “For example, Chandar et al. (2016) propose a hierarchical memory network rather than a ﬂat one and Rae et al. (2016) learn sparse operations to read and write.”
Multi-Level Memory for Task Oriented Dialogs (2018, unknown): “erarchical memory architecture consisting of sentence level memory followed by word memory for a QA task while (Chandar et al., 2016) defines a memory structure that speeds up loading and inferencing over large knowledge bases.”
Contextual Memory Trees (2018, methodology): “Similarly, (Chandar et al., 2016) experiments with a variety of K-MIPS (Maximum Inner Product Search) data structures which the memory tree could potentially replace to create a higher ceiling on performance in situations where MIPS is not the right notion of similarity.”
Neural Machine Translation: A Review (2019, methodology): “s data structures have been used in combination with neural networks such as (inter alia) stacks [422], (double-ended)queues[418], addressablememorycells[417,423,424], andhierarchical memorystructures[425]. Grefenstetteetal.[418]suggestedthatevensimpledatastructures like dequeues help to solve lingu”
On-The-Fly Information Retrieval Augmentation for Language Models (2020, background): “Significant progress in memory networks has been achieved in both architecture (Chandar et al., 2016; Miller et al., 2016; Gulcehre et al., 2017) and model scale (Rae et al.”
Are You Smarter Than a Sixth Grader? Textbook Question Answering for Multimodal Machine Comprehension (2017, methodology): “This suggests the need for either an Information Retrieval based preprocessing step or a hierarchical model such as Hierarchical Memory Networks [5].”
Explore, Propose, and Assemble: An Interpretable Model for Multi-Hop Reading Comprehension (2019, unknown): “2) is a hierarchical memory network (Chandar et al., 2016).”
A^3: Accelerating Attention Mechanisms in Neural Networks with Approximation (2020, methodology): “A few prior works exploit the idea of performing an approximated similarity search for attention mechanisms and implemented them in software [59], [60].”
Multi-Hop Paragraph Retrieval for Open-Domain Question Answering (2019, background): “Many variations of Memory Networks have been proposed, such as end-to-end Memory Networks (Sukhbaatar et al., 2015), Key-Value Memory Networks (Miller et al., 2016), and Hierarchical Memory Networks (Chandar et al., 2016).”
Condensed Memory Networks for Clinical Diagnostic Inferencing (2016, background): “Hierarchical Memory Networks (Chandar et al. 2016) study the use of maximum inner product search (MIPS) to store memory slots in a hierarchy.”
MemexQA: Visual Memex Question Answering (2017, background): “Indeed, an alternative approach is to learn a k-nearest neighbor function from scratch, as suggested in memory networks [4].”
Dynamic Neural Turing Machine with Continuous and Discrete Addressing Schemes (2018, methodology): “Memory networks and their variants have been applied to various tasks successfully (Bordes et al., 2015; Chandar et al., 2016; Dodge et al., 2015; Sukhbaatar et al., 2015; Xiong, Merity, & Socher, 2016).”
\infty-former: Infinite Memory Transformer (2021, background): “…et al. (2014), Weston et al. (2014), Joulin and Mikolov (2015) and Grefenstette et al. (2015) proposed extending RNNs with an external memory, while Chandar et al. (2016) and Rae et al. (2016) proposed efﬁ-cient procedures to read and write from these memories, using hierarchies and sparsity.”