A Simple Method for Commonsense Reasoning

By Trieu H. Trinh, Quoc V. Le. 2018

469 citations recorded by the source corpus.

Commonsense reasoning is a long-standing challenge for deep learning. For example, it is difficult to use neural networks to tackle the Winograd Schema dataset (Levesque et al., 2011). In this paper, we present a simple method for commonsense reasoning with neural networks, using unsupervised learning. Key to our method is the use of language models, trained on a massive amount of unlabled data, to score multiple choice questions posed by commonsense reasoning tests. On both Pronoun Disambiguation and Winograd Schema challenges, our models outperform previous state-of-the-art methods by a large margin, without using expensive annotated knowledge bases or hand-engineered features. We train an array of large RNN language models that operate at word or character level on LM-1-Billion, CommonCrawl, SQuAD, Gutenberg Books, and a customized corpus for this task and show that diversity of training data plays an important role in test performance. Further analysis also shows that our system successfully discovers important features of the context that decide the correct answer, indicating a good grasp of commonsense knowledge.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “• S TORIES , a dataset introduced in Trinh and Le (2018) containing a subset of CommonCrawl data ﬁltered to match the story-like style of Winograd schemas.”
Adversarial Training for Large Neural Language Models (2020, methodology): “For continual pre-training of RoBERTa, we use Wikipedia (13GB), OPENWEBTEXT (public Red-dit content (Gokaslan and Cohen); 38GB), STO-RIES (a subset of CommonCrawl (Trinh and Le, 2018); 31GB).”
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, background): “, 2012) and variants thereof as cloze tasks to investigate commonsense reasoning abilities of language models (Trinh and Le, 2018; Sakaguchi et al., 2019).”
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “We create an aggregate dataset consisting of Wikipedia (Devlin et al., 2018), CC-Stories (Trinh & Le, 2018), RealNews (Zellers et al., 2019), and OpenWeb-text (Radford et al., 2019).”
WinoGrande (2019, methodology): “WSC (Levesque, Davis, and Morgenstern 2011) Liu et al. (2016) 52.8 WKH (Emami et al. 2018) 57.1 Ensemble LMs (Trinh and Le 2018) 63.8 GPT2 (Radford et al. 2019) 70.7 BERT-DPR (cid:63) (Kocijan et al. 2019) 72.2 HNN (He et al. 2019) 75.1 † RoBERTa-DPR (cid:63) (This work) 83.1 RoBERTa-WinoGrande (cid”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, positive): “Language Models 6 (LMs) (Trinh and Le, 2018) trained on large corpora recently achieved the state-of-the-art scores on the Winograd Schema Challenge (Levesque et al., 2011).”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “Yet language models show potential for mitigating these problems directly (Trinh & Le, 2018; Radford et al., 2019) or indirectly (Ra-jani et al., 2019; Xenouleas et al., 2019; Scialom et al., 2019).”
A Hybrid Neural Network Model for Commonsense Reasoning (2019, background): “Trinh and Le (2018) proposed to use a neural language model trained on raw text from books and news to calculate the probabilities of the natural language sentences which are constructed from a statement by replacing the to-be-resolved pronoun in the statement with each of its candidate references…”
A Surprisingly Robust Trick for the Winograd Schema Challenge (2019, methodology): “When using G PT , the probability of a word belonging to the sentence P ( c | s ) is computed as partial loss in the same way as by Trinh and Le (2018).”
One Epoch Is All You Need (2019, background): “Trinh & Le (2018) pointed out that CommonCrawl contains a large portion of corrupt samples, which makes it unsuitable for the training.”
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, methodology): “We also included the CC-Stories and RealNews datasets used to train Megatron [63].”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “• CC-Stories (Trinh and Le, 2018) contains a subset of CommonCrawl data filtered to match the story-like style of Winograd schemas (31GB);”
Solving Quantitative Reasoning Problems with Language Models (2022, methodology): “We find that we can considerably outperform greedy decoding by sampling k > 1 solutions (with a non-zero temperature) and selecting one using majority voting Wang et al. (2022). This consists of grouping predictions”
Big Bird: Transformers for Longer Sequences (2020, methodology): “1 MLM Pretraining We use four publicly available datasets Books [110], CC-News [35], Stories [90] and Wikipedia to pretrain BigBird.”