The LAMBADA dataset: Word prediction requiring a broad discourse context

By Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, R. Fern'andez. 2016

1,160 citations recorded by the source corpus.

We introduce LAMBADA, a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse. We show that LAMBADA exemplifies a wide range of linguistic phenomena, and that none of several state-of-the-art language models reaches accuracy above 1% on this novel benchmark. We thus propose LAMBADA as a challenging test set, meant to encourage the development of new models capable of genuine understanding of broad context in natural language text.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, background): “Last, another related task formulation is sentence completion or cloze, where the task is to predict a single word that is removed from a given context (Zweig and Burges, 2011; Paperno et al., 2016).”
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension (2017, background): “Other recent approaches attempt to combine structured high precision KBs with semi-structured information sources like OpenIE triples (Fader et al., 2014), HTML tables (Pasupat and Liang, 2015), and large (and noisy) corpora (Sawant and Chakrabarti, 2013; Joshi et al., 2014; Xu et al., 2015).”
Training Compute-Optimal Large Language Models (2022, positive): “On LAMBADA (Paperno et al., 2016), Chinchilla outperforms both Gopher and MT-NLG 530B.”
PaLM: Scaling Language Modeling with Pathways (2022, methodology): “…TriviaQA (Joshi et al., 2017), Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) • Cloze and Completion tasks: LAMBADA (Paperno et al., 2016), HellaSwag (Zellers et al., 2019), StoryCloze (Mostafazadeh et al., 2016) • Winograd-style tasks: Winograd (Levesque et…”
Experience Grounds Language (2020, background): “For example, on the LAMBADA dataset (Paperno et al., 2016), designed”
Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, background): “There has been a ﬂurry of work in constructing datasets with an adversarial component, such as Swag (Zellers et al., 2018) and HellaSwag (Zellers et al., 2019), CODAH (Chen et al., 2019), Ad-versarial SQuAD (Jia and Liang, 2017), Lambada (Paperno et al., 2016) and others.”
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “8% overlap and the LAMBADA test set (Paperno et al., 2016) has at most 1.”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, background): “The CNN/Daily Mail Corpus (Hermann et al., 2015), The Children’s Book Test (CBT) (Hill et al., 2015), and LAM-BADA (Paperno et al., 2016) are closely related to Re Co RD : (1) The CNN/Daily Mail Corpus constructed queries from the bullet points, most of which required limited reasoning ability…”
Universal Transformers (2018, methodology): “The LAMBADA task (Paperno et al., 2016) is a language modeling task consisting of predicting a missing target word given a broader context of 4-5 preceding sentences.”
Unified Scaling Laws for Routed Language Models (2022, methodology): “For example, RL-R shows better b(N) for most values of N in LAMBADA, until it is overtaken by S-BASE at N = 410M , but S-BASE is always superior in C4.”
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, background): “LAMBADA The LAMBADA [49] dataset is a collection of narrative passages, speciﬁcally selected such that a human can easily guess the last word if the whole passage is given as context, but would not be able to answer if only given the last sentence in the passage.”
Improving language models by retrieving from trillions of tokens (2021, methodology): “, 2017), Curation Corpus (Curation, 2020), Lambada (Paperno et al., 2016) and the Pile (Gao et al.”
Challenges in Detoxifying Language Models (2021, methodology): “Evaluation on the LAMBADA dataset (Paperno et al., 2016) confirms this trend, with an accuracy decrease from 50.1% to 34.9% for trainfilter@0.05 (Table 7, appendix).”
Compressive Transformers for Long-Range Sequence Modelling (2019, methodology): “Texts extracted from books have been used for prior NLP benchmarks; such as the Children’s Book Test (Hill et al., 2015) and LAMBADA (Paperno et al., 2016).”
PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation (2021, methodology): “Chinese WPLC (Word Prediciton with Long Context) is a dataset created to test the ability to model long-range dependencies, similar to the LAMBADA dataset [34] for English.”