ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning

By Michael Boratko, Xiang Lorraine Li, Rajarshi Das, Timothy J. O'Gorman, Daniel Le, Andrew McCallum. 2020

68 citations recorded by the source corpus.

Given questions regarding some prototypical situation -- such as Name something that people usually do before they leave the house for work? -- a human can easily answer them via acquired experiences. There can be multiple right answers for such questions with some more common for a situation than others. This paper introduces a new question answering dataset for training and evaluating common-sense reasoning capabilities of artificial intelligence systems in such prototypical situations. The training set is gathered from an existing set of questions played in a long-running international trivia game show -- Family Feud. The hidden evaluation set is created by gathering answers for each question from 100 crowd-workers. We also propose an open-domain task where a model has to output a ranked list of answers, ideally covering all prototypical answers for a question. On evaluating our dataset with various competitive state-of-the-art models, we find there is a significant gap between the best model and human performance on a number of evaluation metrics.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…al. 2018 paws cls/paraphrase Zhang et al. 2019 piqa other Bisk et al. 2020 poem_sentiment cls/sentiment analysis Sheng and Uthus 2020 proto_qa other Boratko et al. 2020 qa_srl other He et al. 2015 qasc qa/multiple-choice qa Khot et al. 2020 quail qa/multiple-choice qa Rogers et al. 2020 quarel…”
GraDA: Graph Generative Data Augmentation for Commonsense Reasoning (2022, methodology): “For instance, the question “Besides a mattress, name something people sleep on.” from the ProtoQA dataset (Boratko et al., 2020) can be mapped to the single-hop path (mat-tress, RelatedTo , people) using ConceptNet.”
ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning (2021, background): “…Bisk et al., 2020; Hwang et al., 2020; Forbes et al., 2020), pronoun disambiguation (Sakaguchi et al., 2020; Zhang et al., 2020), abductive commonsense reasoning (Bhagavatula et al., 2019) and general commonsense (Talmor et al., 2019; Huang et al., 2019; Wang et al., 2019; Boratko et al., 2020).”
A Systematic Investigation of Commonsense Understanding in Large Language Models (2021, background): “Various benchmarks have been proposed to test these different types of commonsense knowledge ( e.g. , Zellers et al., 2019b; Sakaguchi et al., 2020; Sap et al., 2019b; Bisk et al., 2020; Lin et al., 2020; Boratko et al., 2020).”
GraDA: Graph Generative Data Augmentation for Commonsense Reasoning (2022, methodology): “Apart from these four MCQ datasets, we also experiment with the generative QA dataset ProtoQA (9762/52/102) (Boratko et al., 2020) and find that our approach works especially well with it.”
Knowledge-Augmented Methods for Natural Language Processing (2023, unknown): “…(Tal-mor et al., 2019), SocialIQA (Sap et al., 2019), PhysicalIQA (Bisk et al., 2020), RiddleSense (Lin et al., 2021b)), 2) open-ended QA (e.g., Pro-toQA (Boratko et al., 2020) OpenCSR (Lin et al., 2021a)), 3) constrained NLG (e.g., Common-Gen (Lin et al., 2020b), conversation generation).”
Reasoning with Transformer-based Models: Deep Learning, but Shallow Reasoning (2021, result): “However, several studies [Forbes et al., 2019, Zhou et al., 2020b, Lin et al., 2020, Boratko et al., 2020, Singh et al., 2021] show that when the datasets are speciﬁcally changed to target the weaknesses of transformer-based models (for example, by adversarial instances), the models fail.”