SituatedQA: Incorporating Extra-Linguistic Contexts into QA

By Michael J.Q. Zhang, Eunsol Choi. 2021

196 citations recorded by the source corpus.

Answers to the same question may change depending on the extra-linguistic contexts (when and where the question was asked). To study this challenge, we introduce SituatedQA, an open-retrieval QA dataset where systems must produce the correct answer to a question given the temporal or geographical context. To construct SituatedQA, we first identify such questions in existing QA datasets. We find that a significant proportion of information seeking questions have context-dependent answers (e.g. roughly 16.5% of NQ-Open). For such context-dependent questions, we then crowdsource alternative contexts and their corresponding answers. Our study shows that existing models struggle with producing answers that are frequently updated or from uncommon locations. We further quantify how existing models, which are trained on data collected in the past, fail to generalize to answering questions asked in the present, even when provided with an updated evidence corpus (a roughly 15 point drop in accuracy). Our analysis suggests that open-retrieval QA benchmarks should incorporate extra-linguistic context to stay relevant globally and in the future. Our data, code, and datasheet are available at https://situatedqa.github.io/.

Subfields: Artificial Intelligence, Information Systems, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Question Answering & Reading Comprehension

Citations in the corpus

LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (2024, unknown): “To more precisely evaluate memory accuracy, subsequent benchmarks shifted toward question answering (QA, Reddy et al. (2019); Zhang & Choi (2021)).”
Entity Cloze By Date: What LMs Know About Unseen Entities (2022, background): “S ITUATED QA (Zhang and Choi, 2021) includes time-dependent QA examples.”
TÜLU 3: Pushing Frontiers in Open Language Model Post-Training (2024, unknown): “Our noncompliance and safety prompts are either curated from existing datasets [Zhang and Choi, 2021, Zhao et al., 2024]”
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models (2022, background): “Furthermore, LMs augmented with retrieval mechanism (Guu et al., 2020; Lewis et al., 2020; Borgeaud et al., 2021) often suffer from hallucination even if they successfully retrieve up-to-date information (Zhang and Choi, 2021; Chen et al., 2021; Longpre et al., 2021).”
Knowledge Conflicts for LLMs: A Survey (2024, unknown): “Previous research has shown that LMs can be significantly influenced by misinformation or outdated information within a specific context (Zhang and Choi, 2021; Du et al., 2022b).”
Generate rather than Retrieve: Large Language Models are Strong Context Generators (2022, background): “As stated in Zhang & Choi (2021), NQ contains a signiﬁcant proportion, roughly 16.5%, of questions that have time-dependent answers.”
ToolQA: A Dataset for LLM Question Answering with External Tools (2023, background): “For example, there are works that study how text retrievers augment LLMs’ performance on open-domain question-answering [21, 74], fact-checking [60], and timely information benchmarks [7, 23, 78, 12].”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, unknown): “[317] create an open-retrieval question answering dataset called SituatedQA, where the systems are required to answer questions given the tempo-spatial contexts.”
RealTime QA: What's the Answer Right Now? (2022, background): “Similar to R EAL T IME QA, Zhang and Choi (2021); Liška et al. (2022) focused on temporal or geographical contexts that can change the answer to the same question.”
StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models (2022, background): “C L ] 2 3 M ay 2 02 2 other datasets and added question dates (Zhang & Choi, 2021) but still answered over a static snapshot of Wikipedia.”
Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2021, background): “, 2021) or political changes, which influence tasks such as question answering (Dhingra et al., 2021; Zhang and Choi, 2021).”
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation (2023, methodology): “Time-sensitive QA : F RESH QA aligns with a growing body of work on benchmarking LLM S ’ temporal reasoning capabilities (Chen et al., 2021b; Zhang & Choi, 2021; Liska et al., 2022; Kasai et al., 2022).”