The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations

By Felix Hill, Antoine Bordes, Sumit Chopra, Jason Aaron Edward Weston. 2015

653 citations recorded by the source corpus.

We introduce a new test of how well language models capture meaning in children's books. Unlike standard language modelling benchmarks, it distinguishes the task of predicting syntactic function words from that of predicting lower-frequency words, which carry greater semantic content. We compare a range of state-of-the-art models, each with a different way of encoding what has been previously read. We show that models which store explicit representations of long-term contexts outperform state-of-the-art neural language models at predicting semantic content words, although this advantage is not observed for syntactic function words. Interestingly, we find that the amount of text encoded in a single memory representation is highly influential to the performance: there is a sweet-spot, not too big and not too small, between single words and full sentences that allows the most meaningful information in a text to be effectively retained and recalled. Further, the attention over such window-based memories can be trained effectively through self-supervision. We then assess the generality of this principle by applying it to the CNN QA benchmark, which involves identifying named entities in paraphrased summaries of news articles, and achieve state-of-the-art performance.

Subfields: Cognitive Neuroscience, Developmental and Educational Psychology, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Developmental and Educational Psychology

In Natural Language Processing

Citations in the corpus

Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering (2018, background): “Among them, CNN/Daily Mail (Hermann et al., 2015) and Children’s Book Test (Hill et al., 2015) are the ﬁrst large-scale datasets for reading comprehension task.”
U-Net: Machine Reading Comprehension with Unanswerable Questions (2018, methodology): “Beneﬁting from the rapid development of deep learning techniques and large-scale benchmarks (Hermann et al. 2015; Hill et al. 2015; Rajpurkar et al. 2016), the end-to-end neural methods have achieved promising results on MRC task (Seo et al. 2016; Huang et al. 2017; Chen et al. 2017; Clark and…”
QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, methodology): “cly available annotated datasets, such as SQuAD (Rajpurkar et al., 2016), TriviaQA (Joshi et al., 2017), CNN/Daily News (Hermann et al., 2015), WikiReading (Hewlett et al., 2016), Children Book Test (Hill et al., 2015), etc. A great number of end-to-end neural network models have been proposed to ta”
Simple and Effective Multi-Paragraph Reading Comprehension (2017, methodology): “The ﬁrst large scale datasets for training neural reading comprehension models used a Cloze-style task, where systems must predict a held out word from a piece of text (Hermann et al., 2015; Hill et al., 2015).”
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension (2017, background): “Researchers have constructed cloze-style datasets (Hill et al., 2015; Hermann et al., 2015; Paperno et al., 2016; Onishi et al., 2016), where the task is to predict missing words, often entities, in a document.”
Bidirectional Attention Flow for Machine Comprehension (2016, methodology): “Inspired by the window-based method (Hill et al., 2016), in order to speed up the training process and reduce GPU memory usage, we only consider words within 7 word-distance of any entity, which we found to be sufficient in most cases.”
SQuAD: 100,000+ Questions for Machine Comprehension of Text (2016, background): “The Children’s Book Test (CBT) (Hill et al., 2015), for example, involves predicting a blanked-out word of a sentence given the 20 previous sentences.”
Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, positive): “…contexts from the following domains: News (extracted from Common Crawl), ﬁction (extracted from Sto-ryCloze (Mostafazadeh et al., 2016) and CBT (Hill et al., 2015)), formal spoken text (excerpted from court and presidential debate transcripts in the Man-ually Annotated Sub-Corpus (MASC) of the…”
Natural Questions: A Benchmark for Question Answering Research (2019, background): “A number of Clozestyle tasks have also been proposed (Hermann et al., 2015; Hill et al., 2015; Paperno et al., 2016; Onishi et al., 2016).”
GLTR: Statistical Detection and Visualization of Generated Text (2019, methodology): “graphs from the bAbI task children book corpus (CBT) (Hill et al., 2015), New York Times articles (NYT), and scientific abstracts from the journals nature and science (SA).”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, background): “ASReader 5 (Kadlec et al., 2016) is a strong base-line model for cloze-style datasets such as (Her-3 mann et al., 2015; Hill et al., 2015).”
CoQA: A Conversational Question Answering Challenge (2018, background): “Instead of simply prepending the current question with its previous questions and answers, Huang et al. (2019) proposed a more sophisticated solution to effectively stack single-turn models along the conversational flow.”
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (2018, methodology): “These datasets have taken various forms: multiple-choice (Richardson et al., 2013), cloze-style (Hermann et al., 2015; Onishi et al., 2016; Hill et al., 2016), and span prediction (Rajpurkar et al., 2016; Trischler et al., 2017; Joshi et al., 2017) However, analysis (Chen et al., 2016; Sug-awara et…”
RACE: Large-scale ReAding Comprehension Dataset From Examinations (2017, methodology): “On CBT, we report performance on two subsets where the missing token is either a common noun (CBT-C) or name entity (CBT-N) since the language models have already reached human-level performance on other types (Hill et al., 2015).”
The LAMBADA dataset: Word prediction requiring a broad discourse context (2016, background): “Another related benchmark, CBT, has been introduced by Hill et al. (2016).”
Matching Networks for One Shot Learning (2016, methodology): “In the CBT for example, a sequence of sentences from a book are provided as context.”
NewsQA: A Machine Comprehension Dataset (2016, methodology): “…2013) are too small for training data-intensive deep learning models, while those that are sufﬁciently large for deep learning (Hermann et al., 2015; Hill et al., 2016; Bajgar et al., 2016) are generated synthetically, yielding questions that are not posed in natural language and that may not…”
Compressive Transformers for Long-Range Sequence Modelling (2019, methodology): “Texts extracted from books have been used for prior NLP benchmarks; such as the Children’s Book Test (Hill et al., 2015) and LAMBADA (Paperno et al., 2016).”
Reformer: The Efficient Transformer (2020, background): “These hints are either given as additional supervising information by the task or determined heuristically as in Hill et al. (2015).”
SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine (2017, background): “One family of such datasets includes Children’s Book dataset (Hill et al., 2015), CNN and DailyMail (Hermann et al., 2015).”