StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models

By Adam Livska, Tom'avs Kovcisk'y, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien de Masson d'Autume, Tim Scholtes, Manzil Zaheer, Susannah Young and 4 more. 2022

140 citations recorded by the source corpus.

Knowledge and language understanding of models evaluated through question answering (QA) has been usually studied on static snapshots of knowledge, like Wikipedia. However, our world is dynamic, evolves over time, and our models' knowledge becomes outdated. To study how semi-parametric QA models and their underlying parametric language models (LMs) adapt to evolving knowledge, we construct a new large-scale dataset, StreamingQA, with human written and generated questions asked on a given date, to be answered from 14 years of time-stamped news articles. We evaluate our models quarterly as they read new articles not seen in pre-training. We show that parametric models can be updated without full retraining, while avoiding catastrophic forgetting. For semi-parametric models, adding new articles into the search space allows for rapid adaptation, however, models with an outdated underlying LM under-perform those with a retrained LM. For questions about higher-frequency named entities, parametric updates are particularly beneficial. In our dynamic world, the StreamingQA dataset enables a more realistic evaluation of QA models, and our experiments highlight several promising directions for future research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Improving alignment of dialogue agents via targeted human judgements (2022, background): “This mechanism also allows for temporal generalisation beyond a static parametric model (Borgeaud et al., 2022; Lewis et al., 2020; Liška et al., 2022; Shuster et al., 2022a).”
Teaching language models to support answers with verified quotes (2022, background): “, 2022) that is kept up-to-date – and supplying them to the language model in a nonparametric fashion – can enable improved temporal generalization over a purely parametric model (Borgeaud et al., 2021; Lewis et al., 2021a; Liska et al., 2022).”
Mass-Editing Memory in a Transformer (2022, background): “…“Polaris is in the constellation Ursa Minor” (Petroni et al., 2020; Brown et al., 2020), even very large models are known to lack more specialized knowledge, and they may recall obsolete information if not updated periodically (Lazaridou et al., 2021; Agarwal & Nenkova, 2022; Liska et al., 2022).”
Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts (2023, methodology): “Tool 1 (Schick et al., 2023; Qin et al., 2023) or retrieval augmentation (Mallen et al., 2022; Shi et al., 2023b; Ram et al., 2023) has emerged as a promising solution by providing external information as new evidence to LLMs, such as ChatGPT Plugins and New Bing.”
Knowledge Conflicts for LLMs: A Survey (2024, unknown): “Retrieved documents may contain updated and outdated information from the network simultaneously, leading to conflicts between these documents (Chen et al., 2021; Liska et al., 2022; Kasai et al., 2022).”
Reading Between the Timelines: RAG for Answering Diachronic Questions (2025, unknown): “To evaluate a model’s ability to adapt to new information, StreamingQA [27] presents questions based on a stream of timestamped news articles.”