UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark

By Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi. 2021

151 citations recorded by the source corpus.

Commonsense AI has long been seen as a near impossible goal---until recently. Now, research interest has sharply increased with an influx of new benchmarks and models. We propose two new ways to evaluate commonsense models, emphasizing their generality on new tasks and building on diverse, recently introduced benchmarks. First, we propose a new multitask benchmark, Rainbow, to promote research on commonsense models that generalize well over multiple tasks and datasets. Second, we propose a novel evaluation, the cost equivalent curve, that sheds new insight on how the choice of source datasets, pretrained language models, and transfer learning methods impacts performance and data efficiency. We perform extensive experiments---over 200 experiments encompassing 4800 models---and report multiple valuable and sometimes surprising findings, e.g., that transfer almost always leads to better or equivalent performance if following a particular recipe, that QA-based commonsense datasets transfer well with each other, while commonsense knowledge graphs do not, and that perhaps counter-intuitively, larger models benefit more from transfer than smaller ones. Last but not least, we introduce a new universal commonsense reasoning model, UNICORN, that establishes new state-of-the-art performance across 8 popular commonsense benchmarks, aNLI (87.3%), CosmosQA (91.8%), HellaSWAG (93.9%), PIQA (90.1%), SocialIQa (83.2%), WinoGrande (86.6%), CycIC (94.0%) and CommonsenseQA (79.3%).

Subfields: Artificial Intelligence, Natural Language Processing, Knowledge Graphs & Representation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Knowledge Graphs & Representation

Citations in the corpus

Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, background): “(2021) for a survey), as well specialized methods to enhance their reasoning abilities (Andor et al., 2019; Ran et al., 2019; Geva et al., 2020; Piękos et al., 2021; Lourie et al., 2021).”
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, methodology): “On common sense reasoning, supervised baseline [38] still outperforms LMs with few-shot learning settings.”
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, methodology): “For the Rainbow suite, we co-train all tasks in (Lourie et al., 2021) and report peak validation results.”
Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, background): “, 2021) or using multitask learning to improve commonsense reasoning (Lourie et al., 2021).”
The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, background): “Rather than competing with the state of the art, which, for many common sense tasks, can be achieved with billion-parameter-scale, closed-book language models (Lourie et al., 2021), we propose a proof-of-concept experimental setup.”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, methodology): “EXT5 outperforms state-of-the-art T5 on well-established benchmarks such as SuperGLUE, GEM, Rainbow, Closed Book Question Answering, and several other tasks that are outside of EXMIX (§3.2), while also being more sample-efficient (§2.6).”
UL2: Unifying Language Learning Paradigms (2022, methodology): “Examples of these include UNICORN (Lourie et al., 2021) for commonsense reasoning, UnifiedQA (Khashabi et al.”
CommonsenseQA 2.0: Exposing the Limits of AI through Gamification (2021, methodology): “4 shows in red 215 the dates in which the model-in-the-loop was trained, as well 216 as the performance (Y-axis) of our best performing baselines, 217 UNICORN [25] and T5 [26], on development set questions dur218 ing those dates.”
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, methodology): “2, GPT-3 models perform substantially worse than humans (>30% less) on SOCIALIQA,4 and also worse than models finetuned on the SOCIALIQA training set (>20%; Lourie et al., 2021).”
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2021, background): “We explore mixing datasets from different NLP benchmarks or families of tasks, including GLUE, SUPERGLUE, natural language inference (NLI), paraphrasing/semantic similarity, sentiment analysis, question answering (QA) on MRQA (Fisch et al., 2019), commonsense reasoning on RAINBOW (Lourie et al., 202”
In-BoXBART: Get Instructions into Biomedical Multi-Task Learning (2022, methodology): “Past attempts have been made in general-domain NLP to achieve generalization across tasks such as MQAN (McCann et al., 2018), UNICORN (Lourie et al., 2021), and UniﬁedQA (Khashabi et al., 2020).”
Reasoning with Transformer-based Models: Deep Learning, but Shallow Reasoning (2021, methodology): “These include data augmentation techniques [Yang et al., 2020], multi-task learning [Lourie et al., 2021], and fusing knowledge graphs into language models [Xu et al., 2021].”
Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering (2022, methodology): “We choose three other UniﬁedQA models with different sizes, as well as a different model known as Unicorn (Lourie et al., 2021).”
LILA: A Unified Benchmark for Mathematical Reasoning (2022, methodology): “We take inspiration from the success of multi-task learning in NLP (Weston et al., 2015), including benchmarks (e.g., Wang et al., 2018, 2019; Dua et al., 2019a) and multitasking models (e.g., McCann et al., 2018; Khashabi et al., 2020; Lourie et al., 2021; Aghajanyan et al., 2021).”
NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks (2022, unknown): “A similar approach is taken by Khashabi et al. (2020) in the setting of question-answering and Lourie et al. (2021) in the scope of commonsense reasoning.”
Generated Knowledge Prompting for Commonsense Reasoning (2021, methodology): “1 For inference, we use off-the-shelf T5 (Raffel et al., 2019) and GPT-3, as well as ﬁnetuned models that are state-of-the-art on each dataset, including UniﬁedQA (UQA) (Khashabi et al., 2020) and Uni-corn (Lourie et al., 2021).”
ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning (2021, background): “While pre-trained language models perform well on these tasks (Lourie et al., 2021), this setup limits the exploration and evaluation of a model’s ability to reason and explain its predictions with relevant commonsense knowledge, thereby allowing models to solve tasks by using shortcuts,…”
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations (2022, methodology): “For supervised models, we consider the strong base-lines used for the respective dataset, such as T5 (Raffel et al., 2020), UniﬁedQA (Khashabi et al., 2020) and Unicorn (Lourie et al., 2021).”