RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms

By Pei Zhou, Rahul Khanna, Bill Yuchen Lin, Daniel E. Ho, Jay Pujara, Xiang Ren. 2020

42 citations recorded by the source corpus.

Pre-trained language models (PTLMs) have achieved impressive performance on commonsense inference benchmarks, but their ability to employ commonsense to make robust inferences, which is crucial for effective communications with humans, is debated. In the pursuit of advancing fluid human-AI communication, we propose a new challenge, RICA: Robust Inference using Commonsense Axioms, that evaluates robust commonsense inference despite textual perturbations. To generate data for this challenge, we develop a systematic and scalable procedure using commonsense knowledge bases and probe PTLMs across two different evaluation settings. Extensive experiments on our generated probe sets with more than 10k statements show that PTLMs perform no better than random guessing on the zero-shot setting, are heavily impacted by statistical biases, and are not robust to perturbation attacks. We also find that fine-tuning on similar statements offer limited gains, as PTLMs still fail to generalize to unseen inferences. Our new large-scale benchmark exposes a significant gap between PTLMs and human-level language understanding and offers a new challenge for PTLMs to demonstrate commonsense.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Lawyers are Dishonest? Quantifying Representational Harms in Commonsense Knowledge Resources (2021, background): “, 2018), probing language models (Petroni et al., 2019; Zhou et al., 2020), question answering (Lin et al.”
Think Before You Speak: Using Self-talk to Generate Implicit Commonsense Knowledge for Response Generation (2021, background): “Recent work also aims to probe models in these tasks to see if reasoning is actually achieved (Richardson and Sabharwal, 2020; Richardson et al., 2020; Zhou et al., 2020; Lin et al., 2021b).”
Knowledge-Augmented Methods for Natural Language Processing (2023, methodology): “Apart from the benchmarking and modeling, we also introduce the analysis works that aim to provide a deeper understanding the common-sense knowledge of pre-trained LMs: LAMA Probing* (Petroni et al., 2019), NumerSense (Lin et al., 2020a), and RICA* (Zhou et al., 2020).”
Commonsense-Focused Dialogues for Response Generation: An Empirical Study (2021, background): “Recent work also aims to probe models in these tasks to see if reasoning is actually achieved (Richardson and Sabharwal, 2020; Richardson et al., 2020; Zhou et al., 2020).”