ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data
By Woojeong Jin, Suji Kim, Rahul Khanna, Dong-Ho Lee, Fred Morstatter, Aram Galstyan, Xiang Ren. 2020
60 citations recorded by the source corpus.
Event forecasting is a challenging, yet important task, as humans seek to constantly plan for the future. Existing automated forecasting studies rely mostly on structured data, such as time-series or event-based knowledge graphs, to help predict future events. In this work, we aim to formulate a task, construct a dataset, and provide benchmarks for developing methods for event forecasting with large volumes of unstructured text data. To simulate the forecasting scenario on temporal news documents, we formulate the problem as a restricted-domain, multiple-choice, question-answering (QA) task. Unlike existing QA tasks, our task limits accessible information, and thus a model has to make a forecasting judgement. To showcase the usefulness of this task formulation, we introduce ForecastQA, a question-answering dataset consisting of 10,392 event forecasting questions, which have been collected and verified via crowdsourcing efforts. We present our experiments on ForecastQA using BERTbased models and find that our best model achieves 61.0% accuracy on the dataset, which still lags behind human performance by about 19%. We hope ForecastQA will support future research efforts in bridging this gap.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jacob Steinhardt
- Xiang Ren
- Percy Liang
- Dan Hendrycks
- Nicholas Carlini
- Nanyun Peng
- Sergey Levine
- Dawn Song
Citations in the corpus
- ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning (2020, background): “…language understanding, and facilitates many real-world applications, such as tracking biomedical histories (Sun et al., 2013; Bethard et al., 2015, 2016, 2017), generating stories (Yao et al., 2019; Goldfarb-Tarrant et al., 2020), and forecasting social events (Li et al., 2020; Jin et al., 2020).”
- Unsolved Problems in ML Safety (2021, background): “To be successful, forecasters must dynamically aggregate information from disparate unstructured sources [94].”
- PromptCast: A New Prompt-Based Learning Paradigm for Time Series Forecasting (2022, background): “Note that the PromptCast QA task differs from recent tasks such as TimeQA [31], which is proposed to answer general time-related questions based on Wikipedia text, and ForecastQA [32], which is also based on text articles.”