PIQA: Reasoning about Physical Commonsense in Natural Language

By Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, Yejin Choi. 2019

3,283 citations recorded by the source corpus.

To apply eyeshadow without a brush, should I use a cotton swab or a toothpick? Questions requiring this kind of physical commonsense pose a challenge to today's natural language understanding systems. While recent pretrained models (such as BERT) have made progress on question answering over more abstract domains – such as news articles and encyclopedia entries, where text is plentiful – in more physical domains, text is inherently limited due to reporting bias. Can AI systems learn to reliably answer physical commonsense questions without experiencing the physical world?In this paper, we introduce the task of physical commonsense reasoning and a corresponding benchmark dataset Physical Interaction: Question Answering or PIQA. Though humans find the dataset easy (95% accuracy), large pretrained models struggle (∼75%). We provide analysis about the dimensions of knowledge that existing models lack, which offers significant opportunities for future research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Training Compute-Optimal Large Language Models (2022, neutral): “We evaluate Chinchilla on various common sense benchmarks: PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), Winogrande (Sakaguchi et al., 2020), HellaSwag (Zellers et al., 2019), and BoolQ (Clark et al., 2019).”
PaLM: Scaling Language Modeling with Pathways (2022, background): “, 2020) • Common Sense Reasoning: PIQA (Bisk et al., 2019), ARC (Clark et al.”
LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “We consider eight standard common sense reasoning benchmarks: BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sak-aguchi et al., 2021), ARC easy and challenge (Clark et al., 2018) and OpenBookQA (Mihaylov et al., 2018).”
Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “We report the average of PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019a), WinoGrande (Sakaguchi et al., 2021), ARC easy and challenge (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), and CommonsenseQA (Talmor et al., 2018).”
Mixtral of Experts (2024, methodology): “We measure performance on a wide variety of tasks categorized as follow: • Commonsense Reasoning (0-shot): Hellaswag [32], Winogrande [26], PIQA [3], SIQA [27], OpenbookQA [22], ARC-Easy, ARC-Challenge [8], CommonsenseQA [30] • World Knowledge (5-shot): NaturalQuestions [20], TriviaQA [19] •…”
UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, positive): “…MCTest (Richard-son et al., 2013), RACE (Lai et al., 2017), OpenBookQA/OBQA (Mihaylov et al., 2018), ARC (Clark et al., 2018, 2016), QASC (Khot et al., 2019), CommonsenseQA/CQA (Talmor et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), and Winogrande (Sakaguchi et al., 2020).”
Experience Grounds Language (2020, background): “from an exponential number of possible how-to, text-only guides and manuals (Bisk et al., 2020) is misdirected without some fundamental referents”
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, background): “For evaluation on PiQA, we prompt the model with the question/goal description and then evaluate the We generally observe minor gain or even performance dips when moving from zero-shot to one-shot, but would observe significant gains when we move from zero-shot to few-shot settings.”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, positive): “In contrast, although fine-tuning of MoE models produces substantial benefits for Storycloze, BoolQ, SST-2, MNLI and some improvements on OpenBookQA, it results in worse performance for HellaSwag, PIQA, and Winogrande.”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, background): “Common Sense Reasoning: PIQA (Bisk et al., 2020), ARC (Easy) (Clark et al.”
Measuring Massive Multitask Language Understanding (2020, background): “While these benchmarks evaluate linguistic skills more than overall language understanding, an array of commonsense benchmarks have been proposed to measure basic reasoning and everyday knowledge (Zellers et al., 2019; Huang et al., 2019; Bisk et al., 2019).”
Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “PiQA (Bisk et al., 2020) is a commonsense QA benchmark for naive physics reasoning, where a solution to a goal must be selected from two choices.”
OPT: Open Pre-trained Transformer Language Models (2022, methodology): “We evaluate our model on 16 standard NLP tasks utilized in the literature: HellaSwag (Zellers et al., 2019), StoryCloze (Mostafazadeh et al., 2016), PIQA (Bisk et al., 2020), ARC Easy and Challenge (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), WinoGrad (Levesque et al., 2011), Wino-Grand”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “Due to space constraints a representative subset of the results are shown here, with the rest in Appendix E. Natural Language Tasks We evaluate our model on a diverse collection of standard language model evaluation datasets: ANLI (Nie et al., 2020), ARC (Clark et al., 2018), HeadQA (English) (Vilar”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “(Gao et al., 2020; Rae et al., 2021), we found WET files to include undesirable navigation menus, ads, and other irrelevant texts.”
How Context Affects Language Models' Factual Predictions (2020, background): “…later extends LAMA by automatically discovering better prompts, Kassner and Sch¨utze [2019] add negated statements, Poerner et al. [2019] ﬁlter out easy-to-guess queries, and Richardson and Sabharwal [2019], Talmor et al. [2019], Bisk et al. [2019] develop further probes for textual reasoning.”
Natural Adversarial Examples (2019, background): “Many recently introduced NLP datasets use adversarial filtration to create “adversarial datasets” by sieving examples solved with simple spurious cues [44, 4, 56, 13, 6, 25].”