RealTime QA: What's the Answer Right Now?

By Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Velocity Yu, Dragomir R. Radev, Noah A. Smith, Yejin Choi, Kentaro Inui. 2022

301 citations recorded by the source corpus.

We introduce REALTIME QA, a dynamic question answering (QA) platform that announces questions and evaluates systems on a regular basis (weekly in this version). REALTIME QA inquires about the current world, and QA systems need to answer questions about novel events or information. It therefore challenges static, conventional assumptions in open-domain QA datasets and pursues instantaneous applications. We build strong baseline models upon large pretrained language models, including GPT-3 and T5. Our benchmark is an ongoing effort, and this paper presents real-time evaluation results over the past year. Our experimental results show that GPT-3 can often properly update its generation results, based on newly-retrieved documents, highlighting the importance of up-to-date information retrieval. Nonetheless, we find that GPT-3 tends to return outdated answers when retrieved documents do not provide sufficient information to find an answer. This suggests an important avenue for future research: can an open-domain QA system identify such unanswerable cases and communicate with the user or even the retrieval module to modify the retrieval results? We hope that REALTIME QA will spur progress in instantaneous applications of question answering and beyond.

Subfields: Artificial Intelligence, Information Systems, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Question Answering & Reading Comprehension

Citations in the corpus

Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2024, background): “This approach mitigates issues with frequent manual updates required for maintaining previous datasets (Vu et al., 2023; Kasai et al., 2024).”
Measuring and Narrowing the Compositionality Gap in Language Models (2022, methodology): “Therefore, we can use a search engine to answer the sub-questions instead of the LM. Search engines have features that LMs lack, such as an ability to be easily and quickly updated (Kasai et al., 2022).”
Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity (2024, background): “However, they still generate factually incorrect answers since their knowledge solely relies on their parametric memory (Kasai et al., 2022; Mallen et al., 2023).”
Ask Me Anything: A simple strategy for prompting language models (2022, methodology): “…Classification (DBPedia [Zhang et al., 2015], AGNews [Zhang et al., 2015], SST2 [Socher et al., 2013], Amazon [He and McAuley, 2016]), and Question-Answering (RealTimeQA [Kasai et al., 2022], DROP [Dua et al., 2019], Natural Questions [Kwiatkowski et al., 2019], WebQuestions [Berant et al., 2013]).”
Retrieval-based Language Models and Applications (2023, background): “Moreover, these parametric LMs are fundamentally incapable of adapting over time (De Cao et al., 2021; Lazaridou et al., 2021; Kasai et al., 2022), often hallucinate (Shus-ter et al., 2021), and may leak private data from the training corpus (Carlini et al., 2021).”
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation (2023, unknown): “Figure 7 in Appendix G shows a realistic prompt.”
When Not to Trust Language Models: Investigating Effectiveness and Limitations of Parametric and Non-Parametric Memories (2022, background): “, 2020) and the knowledge can get obsolete quickly (Kasai et al., 2022; Jang et al., 2022).”
ToolQA: A Dataset for LLM Question Answering with External Tools (2023, background): “For example, there are works that study how text retrievers augment LLMs’ performance on open-domain question-answering [21, 74], fact-checking [60], and timely information benchmarks [7, 23, 78, 12].”
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (2022, background): “However, for many open-domain questions, all required knowledge is not always available or up-todate in models’ parameters and it’s beneficial to retrieve knowledge from external sources (Lazaridou et al., 2022; Kasai et al., 2022).”
Knowledge Conflicts for LLMs: A Survey (2024, unknown): “Given the implications for the trustworthiness (Du et al., 2022b), real-time accuracy (Kasai et al., 2022), and robustness (Ying et al., 2023) of LLMs, it is imperative to delve deeper into understanding such conflicts (Xie et al., 2023; Wang et al., 2023e).”
When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (2022, background): “Meanwhile, LMs have also been shown to have limited memorization for less frequent entities (Kandpal et al., 2022), are prone to hallucinations (Shuster et al., 2021), and suffer from temporal degradation (Kasai et al., 2022; Jang et al., 2022).”
Tool Learning with Foundation Models (2023, unknown): “We experiment with RealTimeQA (Kasai et al., 2022), which is a dynamic question-answering platform that inquires about novel events or information.”
Prompting GPT-3 To Be Reliable (2022, unknown): “For example, LLMs (including GPT-3) produce biased (Lucy & Bamman, 2021) generations, false statements (Lin et al., 2022b), and outdated information (Chen et al., 2021b; Kasai et al., 2022).”
Evaluation of Retrieval-Augmented Generation: A Survey (2024, background): “Retrieval Metrics Various targets can be evaluated with various metrics that correspond to the given datasets.”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, unknown): “The factual knowledge embedded within LLMs exhibits clear temporal boundaries and can become outdated over time (Onoe et al., 2022; Kasai et al., 2022; Li et al., 2023a).”
FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents (2025, unknown): “We extract relevant posts and answers from the Stack Overflow XML data archive (dated October 2024).”