Weakly Supervised Memory Networks

By Sainbayar Sukhbaatar, Arthur Szlam, Jason Aaron Edward Weston, Rob Fergus. 2015

107 citations recorded by the source corpus.

In this paper we introduce a variant of Memory Networks that needs significantly less supervision to perform question and answering tasks. The original model requires that the sentences supporting the answer be explicitly indicated during training. In contrast, our approach only requires the answer to the question during training. We apply the model to the synthetic bAbI tasks, showing that our approach is competitive with the supervised approach, particularly when trained on a sufficiently large amount of data. Furthermore, it decisively beats other weakly supervised approaches based on LSTMs. The approach is quite general and can potentially be applied to many other tasks that require capturing long-term dependencies.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Attention-Based Models for Speech Recognition (2015, methodology): “[18] describes a similar hybrid attention mechanism, where location embeddings are used as input to the attention model.”
Large-scale Simple Question Answering with Memory Networks (2015, methodology): “Second, in sections 3 and 4, we present an embedding-based QA system developed under the framework of Memory Networks (MemNNs) (Weston et al., 2015; Sukhbaatar et al., 2015).”
Learning to Transduce with Unbounded Memory (2015, methodology): “Finally, to study the effect of random initialisation, we furthermore re-ran the experiments using random seeds in [1 , 12] , which were used to initialise the random number generator at the start of training.”
Visual Madlibs: Fill in the blank Image Generation and Question Answering (2015, background): “Recently, embedding and deep learning methods have shown great promise for question-answering [30, 3, 4].”
Hierarchical Attention Networks for Document Classification (2016, background): “The context vector u w can be seen as a high level representation of a ﬁxed query “what is the informative word” over the words like that used in memory networks (Sukhbaatar et al., 2015; Kumar et al., 2015).”
Conversational AI: The Science Behind the Alexa Prize (2018, methodology): “, 2017), on the other hand, used Dynamic Memory Networks (Sukhbaatar et al., 2015), Character-Level Recursive Neural Networks (RNN) (Sutskever et al.”
Reinforcement Learning Neural Turing Machines (2015, methodology): “Sukhbaatar et al. [14] addressed this problem using differentiable attention within the Memory Network framework.”
Learning to Remember Rare Events (2017, background): “Sukhbaatar et al. (2015) present a similar augmentation and show the importance of allowing multiple reads and writes to memory between inputs.”
Where to Look: Focus Regions for Visual Question Answering (2015, methodology): “isting object annotations into questions, which provides an interesting visual task but limits the scope of visual and abstract knowledge required. Our model is inspired by End-to-End Memory Networks [19] proposed for answering questions based on a series of sentences. The regions in our model are a”
Describing Multimedia Content Using Attention-Based Encoder-Decoder Networks (2015, methodology): “, (14)] to find relevant memory contents, which was futher extended to the weakly supervised memory network in [66] in Section IV-E3.”
Learning Simple Algorithms from Examples (2015, background): “End-to-End Memory Networks (Sukhbaatar et al., 2015) use a feed-forward network as the controller and interfaces consisting of a soft-attention input, plus a delayed output (by a fixed number of “hops”).”
Machine Comprehension with Syntax, Frames, and Semantics (2015, methodology): “There is also recent work on QA based on synthetic text describing events in adventure games (Weston et al., 2015; Sukhbaatar et al., 2015).”
Towards Neural Network-based Reasoning (2015, methodology): “The results of Memory Net-step, Memory Net-N2N, and Dynamic Memory Net are taken respectively from [9],[6] and [5].”
Visual question answering: A survey of methods and datasets (2016, background): “They were described in [42], with a number of variants proposed concurrently [83, 70, 8, 59].”
Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems (2015, background): “Attention can be seen as analogous to the “soft addressing” mechanisms of the recently proposed Neural Turing Machine (Graves et al., 2014) and End-To-End Memory Network (Sukhbaatar et al., 2015) models.”
Reinforcement Learning Neural Turing Machines - Revised (2015, unknown): “Weakly supervised Memory Network (Sukhbaatar et al., 2015) uses a feed forward network as the Controller, and has a differentiable soft-attention Input, and Delayed Output as Interfaces.”
Personalization in Goal-Oriented Dialog (2017, background): “…mechanisms for neural networks architectures have led to remarkable progress in machine translation (Bahdanau et al., 2014; Johnson et al., 2016), question answering (Sukhbaatar et al. 2015, Graves et al., 2016) and other language understanding tasks which require an element of logical reasoning.”
Memorization Precedes Generation: Learning Unsupervised GANs with Memory Networks (2018, background): “Augmenting neural networks with memory has been studied much (Bahdanau et al., 2014; Graves et al., 2014; Sukhbaatar et al., 2015).”