CommonsenseQA 2.0: Exposing the Limits of AI through Gamification

By Alon Talmor, Ori Yoran, Ronan Le Bras, Chandrasekhar Bhagavatula, Yoav Goldberg, Yejin Choi, Jonathan Berant. 2021

177 citations recorded by the source corpus.

Constructing benchmarks that test the abilities of modern natural language understanding models is difficult - pre-trained language models exploit artifacts in benchmarks to achieve human parity, but still fail on adversarial examples and make errors that demonstrate a lack of common sense. In this work, we propose gamification as a framework for data construction. The goal of players in the game is to compose questions that mislead a rival AI while using specific phrases for extra points. The game environment leads to enhanced user engagement and simultaneously gives the game designer control over the collected data, allowing us to collect high-quality data at scale. Using our method we create CommonsenseQA 2.0, which includes 14,343 yes/no questions, and demonstrate its difficulty for models that are orders-of-magnitude larger than the AI used in the game itself. Our best baseline, the T5-based Unicorn with 11B parameters achieves an accuracy of 70.2%, substantially higher than GPT-3 (52.9%) in a few-shot inference setup. Both score well below human performance which is at 94.1%.

Subfields: Computer Science Applications, Computer Vision and Pattern Recognition, Safety Research, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Computer Vision and Pattern Recognition

In Safety Research

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, background): “Commonsense reasoning is key for interacting with the world and is still beyond the reach of current natural language understanding systems (Talmor et al., 2021).”
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2021, background): “Prior works have suggested that pre-trained language models possess limited understanding of commonsense knowledge (Merrill et al., 2021; Talmor et al., 2021; Davis and Marcus, 2017) despite (1)We will share this following the anonymity period.”
Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits (2023, background): “Current large-scale pre-trained language models (LMs) have shown great success in many knowledge-recalling tasks, such as question answering [62] and entity retrieval [12]; however, their ability to select socially good text from bad (or generating prosocial text) in open-world settings is still lim”
Unifying Language Learning Paradigms (2022, unknown): “We use HellaSwag (Zellers et al., 2019), SocialIQA/SIQA (Sap et al., 2019), PhysicalIQA/PIQA (Bisk et al., 2020), CosmosQA (Huang et al., 2019), AbductiveNLI (Bhagavatula et al., 2019), CommonsenseQA (Talmor et al., 2018), Com-monsenseQA2 (Talmor et al., 2021).”
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations (2022, methodology): “We prompt 175B GPT-3 with 100 questions sampled from Talmor et al. (2021). lem (Wei et al., 2022), or the intermediate steps of program execution (Nye et al., 2021a).”
Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs (2023, methodology): “As shown in Figure 2 (b), we employ a double-blind approach [42] for SQL annotation.”
Generated Knowledge Prompting for Commonsense Reasoning (2021, methodology): “For CSQA2, we use snippets returned by Google when querying the question.”