How Additional Knowledge can Improve Natural Language Commonsense Question Answering

By Arindam Mitra, Pratyay Banerjee, Kuntal Kumar Pal, Swaroop Mishra, Chitta Baral. 2019

32 citations recorded by the source corpus.

Recently several datasets have been proposed to encourage research in Question Answering domains where commonsense knowledge is expected to play an important role. Recent language models such as ROBERTA, BERT and GPT that have been pre-trained on Wikipedia articles and books have shown reasonable performance with little fine-tuning on several such Multiple Choice Question-Answering (MCQ) datasets. Our goal in this work is to develop methods to incorporate additional (commonsense) knowledge into language model-based approaches for better question-answering in such domains. In this work, we first categorize external knowledge sources, and show performance does improve on using such sources. We then explore three different strategies for knowledge incorporation and four different models for question-answering using external commonsense knowledge. We analyze our predictions to explore the scope of further improvements.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, neutral): “(Mitra et al., 2020) RoBERTa (Clark et al.,2019c) FreeLB-RoBERTa (Zhu et al., 2020) RoBERTa (Clark et al.,2019c) FreeLB-RoBERTa (Zhu et al., 2020) -- KF+SIR +2Step (Mitra et al., 2020) 75.7 (Lan et al.,2019) FreeLB-RoBERTa (Zhu et al.,2020) RoBERTa (Sakaguchi et al.,2019) RoBERTa (Bisk et al.,…”
The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, background): “…NLP tasks, while large pretrained models have achieved remarkable performance (Raffel et al., 2019; Brown et al., 2020; Lourie et al., 2021), researchers have been seeking external repositories of common sense to boost performance further (Mitra et al., 2019; Lin et al., 2021a; Xu et al., 2021).”
Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering (2022, methodology): “Yet these methods either retrieve from indomain knowledge bases (Mitra et al., 2019; Chang et al., 2020) that do not have good coverage over commonsense, or generate knowledge from neural models (Shwartz et al.”
Knowledge Fusion and Semantic Knowledge Ranking for Open Domain Question Answering (2020, methodology): “External Knowledge: Closest to our work are the models which use external knowledge, such as in the work of (Khot et al., 2019b,a; Pirtoaca et al., 2019; Banerjee et al., 2019; Mitra et al., 2019a) where they use sentences as knowledge.”
Incorporating Commonsense Knowledge Graph in Pretrained Models for Social Commonsense Tasks (2020, methodology): “When using ATOMIC, because there are some blanks and unknown names such as ”PersonX meets for lunch”, we replace PersonX and PersonY with two different common last names to avoid gender bias, and following Mitra et al. (2019), we utilize the pretrained BERT-large’s MLM head to fill in the blanks.”
Kformer: Knowledge Injection in Transformer Feed-Forward Layers (2022, methodology): “RoBERTa+MCQueen [12] explored different ways to infuse knowledge into the language model, including concat, parallelmax, simple sum and weighted sum.”
Generated Knowledge Prompting for Commonsense Reasoning (2021, methodology): “One approach is to inject commonsense knowledge into language models, either by pretraining on knowledge bases (Ma et al., 2021; Chang et al., 2020; Mitra et al., 2019; Zhong et al., 2019) or ﬁnetuning the model so that it can reason with additional retrieved knowledge (Chang et al., 2020; Mitra et…”