CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning

By Bill Yuchen Lin, Ming Shen, Wangchunshu Zhou, Pei Zhou, Chandra Bhagavatula, Yejin Choi, Xiang Ren. 2020

417 citations recorded by the source corpus.

Recently, large-scale pre-trained language models have demonstrated impressive performance on several commonsense-reasoning benchmark datasets. However, building machines with commonsense to compose realistically plausible sentences remains challenging. In this paper, we present a constrained text generation task, CommonGen associated with a benchmark dataset, to explicitly test machines for the ability of generative commonsense reasoning. Given a set of common concepts (e.g., dog, frisbee, catch, throw); the task is to generate a coherent sentence describing an everyday scenario using these concepts (e.g., “a man throws a frisbee and his dog catches it”). The CommonGen task is challenging because it inherently requires 1) relational reasoning with background commonsense knowledge and 2) compositional generalization ability to work on unseen concept combinations. Our dataset, constructed through a combination of crowdsourced and existing caption corpora, consists of 77k commonsense descriptions over 35k unique concept-sets. Experiments show that there is a large gap between state-of-the-art text generation models (e.g., T5) and human performance (31.6% v.s. 63.5% in SPICE metric). Furthermore, we demonstrate that the learned generative commonsense reasoning capability can be transferred to improve downstream tasks such as CommonsenseQA (76.9% to 78.4 in dev accuracy) by generating additional context.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Finetuned Language Models Are Zero-Shot Learners (2021, neutral): “CommonGen (Lin et al., 2020) 2.”
Self-Refine: Iterative Refinement with Self-Feedback (2023, methodology): “Dataset: (Lin et al., 2020) 200 samples x: beach, vacation, relaxation yt: During our beach vacation.”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, neutral): “– GEM (en): Wiki-Lingua (Faisal Ladhak & McKeown, 2020), WenNLG (Gardent et al., 2017; Castro Ferreira et al., 2020), CommonGEN (Lin et al., 2020), E2E (Dušek et al., 2019), DART (Radev et al., 2020), ToTTo (Parikh et al., 2020), Wiki-Auto (Jiang et al., 2020), TurkCorpus(Xu et al., 2016) –…”
Memory-assisted prompt editing to improve GPT-3 after deployment (2022, background): “Examples for usage in a sentence (sent) are from Commongen (Lin et al., 2020).”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “CommonGEN (Lin et al., 2020) Generative Commonsense Concept Set A sentence coherently using all input concepts.”
UL2: Unifying Language Learning Paradigms (2022, methodology): “For WebNLG, E2E and CommonGen, we use the versions from the GEM benchmark (Gehrmann et al., 2021).”
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021, background): “tence pairs (filtered by edit distance to guarantee closeness) in non-paired datasets including CommonGen (Lin et al., 2020), Natural Questions (Kwiatkowski et al.”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…McCreery et al. 2020 mocha other/regression Chen et al. 2020a multi_news cg/summarization Fabbri et al. 2019 numer_sense qa/closed-book qa Lin et al. 2020a onestop_english cls/other Vajjala and Lučić 2018 openbookqa qa/multiple-choice qa Mihaylov et al. 2018 paws cls/paraphrase Zhang…”
Language Models are General-Purpose Interfaces (2022, background): “• Struct to Text is to construct a natural language description for some structured data: Common-Gen (Lin et al., 2020), E2ENLG (Dušek et al., 2020).”
A Survey on Data Augmentation for Text Classification (2021, neutral): “[129] consider another filtering mechanism in their work that does not need any human assistance and is sophisticated by incorporating two perspectives.”
Pre-training Text-to-Text Transformers for Concept-centric Common Sense (2020, methodology): “, 2016) following the leaderboard of COMMONGEN (Lin et al., 2020).”
COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics (2022, positive): “For example (Figure 1), for knowledge-grounded or keyword-guided generation, we might want to ensure that certain keywords are included in the generated output as hard lexical constraints [29, 52].”
Gradient-based Constrained Sampling from Language Models (2022, neutral): “In C OMMON G EN (Lin et al., 2020) given no prompt , the task is generate an output of maximum length 40 which contains a given set of four or five words.”
A Systematic Investigation of Commonsense Understanding in Large Language Models (2021, background): “Various benchmarks have been proposed to test these different types of commonsense knowledge ( e.g. , Zellers et al., 2019b; Sakaguchi et al., 2020; Sap et al., 2019b; Bisk et al., 2020; Lin et al., 2020; Boratko et al., 2020).”