A Hybrid Neural Network Model for Commonsense Reasoning

By Pengcheng He, Xiaodong Liu, Weizhu Chen, Jianfeng Gao. 2019

30 citations recorded by the source corpus.

This paper proposes a hybrid neural network(HNN) model for commonsense reasoning. An HNN consists of two component models, a masked language model and a semantic similarity model, which share a BERTbased contextual encoder but use different model-specific input and output layers. HNN obtains new state-of-the-art results on three classic commonsense reasoning tasks, pushing the WNLI benchmark to 89%, the Winograd Schema Challenge (WSC) benchmark to 75.1%, and the PDP60 benchmark to 90.0%. An ablation study shows that language models and semantic similarity models are complementary approaches to commonsense reasoning, and HNN effectively combines the strengths of both. The code and pre-trained models will be publicly available at https: //github.com/namisan/mt-dnn.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yejin Choi
2. Jianfeng Gao
3. Weizhu Chen
4. Xiaodong Liu
5. Mohit Bansal
6. Ronan Le Bras
7. Hannaneh Hajishirzi
8. Pengcheng He

In Molecular Biology

1. Ronan Le Bras
2. Chandra Bhagavatula
3. Jonathan Pilault
4. Yejin Choi
5. Jianfeng Gao
6. C. Pal
7. Xiaodong Liu
8. Weizhu Chen

In Natural Language Processing

1. Yejin Choi
2. Jianfeng Gao
3. Xiaodong Liu
4. Weizhu Chen
5. Ronan Le Bras
6. Chandra Bhagavatula
7. Pengcheng He
8. Mohit Bansal

Citations in the corpus

• WinoGrande (2019, methodology): “While humans find W INO G RANDE problems trivial with 94% accuracy, best state-of-the-art results, including those from RoBERTa (Liu et al. 2019) are considerably lower, ranging between 59.4% - 79.1% depending on the amount of training data provided (from 800 to 41k instances), which falls 15 - 35%…”
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention (2020, methodology): “These PLMs have been fine-tuned using task-specific labels and created new state-of-the-art in many downstream natural language processing (NLP) tasks [13, 14, 15, 16, 17, 18].”
• Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data (2020, methodology): “Unlike MT-DNNSMART however, we do not add context from a semantic similarity model – MT-DNNSMART is built off HNN (He et al., 2019).”
• Contrastive Self-Supervised Learning for Commonsense Reasoning (2020, background): “…and after the success of natural language representation learning, several works proposed to use supervised learning to discover commonsense relationships, achieving state-of-the-art in multiple benchmarks (see, e.g., (Kocijan et al., 2019; He et al., 2019; Ye et al., 2019; Ruan et al., 2019)).”