Improving Question Answering by Commonsense-Based Pre-Training
By Wanjun Zhong, Duyu Tang, Nan Duan, Ming Zhou, Jiahai Wang, Jian Yin. 2018
489 citations recorded by the source corpus.
Although neural network approaches achieve remarkable success on a variety of NLP tasks, many of them struggle to answer questions that require commonsense knowledge. We believe the main reason is the lack of commonsense \mbox{connections} between concepts. To remedy this, we provide a simple and effective method that leverages external commonsense knowledge base such as ConceptNet. We pre-train direct and indirect relational functions between concepts, and show that these pre-trained functions could be easily added to existing neural network models. Results show that incorporating commonsense-based function improves the baseline on three question answering tasks that require commonsense reasoning. Further analysis shows that our system \mbox{discovers} and leverages useful evidence from an external commonsense knowledge base, which is missing in existing neural network models and help derive the correct answer.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Xiangnan He
- Tat-seng Chua
- Xiang Wang
- Zhiyuan Liu
- Wayne Xin Zhao
- Yongfeng Zhang
- Philip S. Yu
- Meng Wang
In Computer Vision and Pattern Recognition
In Question Answering & Reading Comprehension
Citations in the corpus
- Improving Question Answering with External Knowledge (2019, neutral): “, (Khot et al., 2017; Zhang et al., 2018; Zhong et al., 2018)) explore many ways of exploiting structured knowledge.”
- Explain Yourself! Leveraging Language Models for Commonsense Reasoning (2019, neutral): “Commonsense reasoning is a challenging task for modern machine learning methods (Zhong et al., 2018; Talmor et al., 2019).”
- ERNIE: Enhanced Language Representation with Informative Entities (2019, positive): “…embeddings and entity embeddings for the next layer. significantly enhance original models, such as reading comprehension (Mihaylov and Frank, 2018; Zhong et al., 2018), machine translation (Zaremoodi et al., 2018), natural language inference (Chen et al., 2018), knowledge acquisition (Han et…”
- Neural Input Search for Large Scale Recommendation Models (2019, methodology): “For the MovieLens-1M dataset, since it is formulated as a retrieval-type problem with sampled negatives, we used Recall@1 (also called HitRate@1 in some literatures [24, 26])”
- Empowering Things With Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things (2020, background): “These application scenarios could benefit from recent advances in causal inference and discovery [39], graph-based reasoning [40], reinforcement learning (RL) [41], and speech recognition and synthesis [24], [42].”
- ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning (2020, unknown): “(Zhong et al., 2019; Wang et al., 2019) This makes it a good potential source for solving this task, as well as assessing how well this dataset captures existing notions of common sense.”
- Improving Conversational Recommender Systems via Knowledge Graph based Semantic Fusion (2020, background): “Especially, knowledge graphs have been widely adopted to enhance the recommendation performance and explainability [10, 31, 37].”
- Neural Graph Collaborative Filtering (2019, methodology): “To enhance the embedding function, much effort has been devoted to incorporate side information like item content [2, 29], social relations [32], item relations [35], user reviews [3], and external knowledge graph [30, 33].”
- Personalized Reason Generation for Explainable Song Recommendation (2019, background): “[42] propose a new perspective that leverages knowledge graphs for explainable recommendation.”
- A Survey on Knowledge Graphs: Representation, Acquisition, and Applications (2020, background): “While other works consider the relational path and structure of knowledge graphs, KPRN [186] regards the interaction between users and items as an entity-relation path in the knowledge graph and conducts preference inference over the path with LSTM to capture the sequential dependency.”
- Multi-Channel Graph Neural Network for Entity Alignment (2019, methodology): “Inspired by Wang et al. (2018b), which utilize Graph Convolutional Network (GCN) (Kipf and Welling, 2017) to encode the entire KGs, we aim at reconciling the heterogeneity between KGs through completion and pruning, and learn alignment-oriented KG embeddings by modeling structural features from…”
- A Survey of Graph Neural Networks for Recommender Systems: Challenges, Methods, and Directions (2021, methodology): “Moreover, considering its success in scenarios, such as sequential recommendation [75, 185], it is promising that the leverage of KG could further enhance recommendation quality from the aspects that require more external knowledge, such as diversity and fairness.”
- Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering (2019, unknown): “Zhong et al. (2018) proposed to pre-train a scoring function using knowledge triples from ConceptNet, to model the direct and indirect relation between concepts.”
- Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension (2019, unknown): “Zhong et al. (2018) try to leverage pre-trained KB embeddings to solve the multi-choice MRC task.”
- KGAT: Knowledge Graph Attention Network for Recommendation (2019, background): “It suggests that considering third-order relations among entities could be sufficient to capture the collaborative signal, which is consistent to the findings in [14, 33].”
- KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning (2019, unknown): “CBPT (Zhong et al., 2018) is a plug-in method of assembling the predictions of any models with a straightforward method of utilizing pre-trained concept embeddings from ConceptNet.”
- Generated Knowledge Prompting for Commonsense Reasoning (2021, unknown): “One approach is injecting commonsense knowledge into language models, either by pretraining on knowledge bases (Ma et al., 2021; Chang et al., 2020; Mitra et al., 2019; Zhong et al., 2019) or finetuning the model so that it can reason with additional knowledge retrieved from the knowledge bases (Cha”
- Learning to Attend On Essential Terms: An Enhanced Retriever-Reader Model for Open-domain Question Answering (2018, unknown): “70 / / TriAN + f(dir)(cs) + f(ind)(cs) (Zhong et al., 2018) 33.”