An End-to-End Model for Question Answering over Knowledge Base with Cross-Attention Combining Global Knowledge

By Yanchao Hao, Yuanzhe Zhang, Kang Liu, Shizhu He, Zhanyi Liu, Hua Wu, Jun Zhao. Annual Meeting of the Association for Computational Linguistics, 2017

370 citations recorded by the source corpus.

With the rapid growth of knowledge bases (KBs) on the web, how to take full advantage of them becomes increasingly important. Question answering over knowledge base (KB-QA) is one of the promising approaches to access the substantial knowledge. Meanwhile, as the neural network-based (NN-based) methods develop, NN-based KB-QA has already achieved impressive results. However, previous work did not put more emphasis on question representation, and the question is converted into a fixed vector regardless of its candidate answers. This simple representation strategy is not easy to express the proper information in the question. Hence, we present an end-to-end neural network model to represent the questions and their corresponding scores dynamically according to the various candidate answer aspects via cross-attention mechanism. In addition, we leverage the global knowledge inside the underlying KB, aiming at integrating the rich KB information into the representation of the answers. As a result, it could alleviates the out-of-vocabulary (OOV) problem, which helps the cross-attention model to represent the question more precisely. The experimental results on WebQuestions demonstrate the effectiveness of the proposed approach.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/e9287b896a1c7360567915c3932b8df1ee4a81f7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xiangnan He
2. Tat-seng Chua
3. Philip S. Yu
4. Xiang Wang
5. Zhiyuan Liu
6. Erik Cambria
7. Maosong Sun
8. Mohit Bansal

In Information Systems

1. Xiangnan He
2. Xiang Wang
3. Tat-seng Chua
4. Chuan Shi
5. Wayne Xin Zhao
6. Yixin Cao
7. Philip S. Yu
8. Maosong Sun

Citations in the corpus

• Sparse, Dense, and Attentional Representations for Text Retrieval (2020, neutral): “The cross attention inner product is defined as, ψ(X)(x, y) = Tx∑ t=1 〈 xt, Ty∑ t′=1 ax,y(t, t ′)× yt′ 〉 , (6) where ax,y(t, t′) ∈ R+ is the attention from token xt to yt′ (Yang et al., 2016; Hao et al., 2017).”
• Knowledge Graph Embedding Based Question Answering (2019, background): “Several deep learning based models [10, 12, 21, 27, 46] achieved this projection by feeding words in questions into convolutional neural networks [12, 46], LSTM networks [18, 21], or gated recurrent units neural networks [10, 27].”
• Knowledge Aware Emotion Recognition in Textual Conversations via Multi-Task Incremental Transformer (2020, background): “…has grown up around the theme of incorporating external knowledge in generative conversation systems, including question answering systems (Hao et al., 2017; Mihaylov and Frank, 2018), open-domain dialogue systems (Young et al., 2018; Zhou et al., 2018b; Zhong et al., 2019a), and…”
• Leveraging Meta-path based Context for Top- N Recommendation with A Neural Co-Attention Model (2018, background): “In particular, co-attention or crossattention mechanisms have been applied to solve complicated NLP tasks [8, 36].”
• Leveraging Domain Context for Question Answering Over Knowledge Graph (2019, methodology): “[13] discussed attention-based end-to-end model with the global knowledge graph features for question answering.”
• Knowledge-Enriched Transformer for Emotion Detection in Textual Conversations (2019, background): “…Pineau, 2018; Liu et al., 2018; Moghe et al., 2018; Dinan et al., 2019; Zhong et al., 2019), task-oriented dialogue systems (Madotto et al., 2018; Wu et al., 2019; He et al., 2019) and question answering systems (Kiddon et al., 2016; Hao et al., 2017; Sun et al., 2018; Mihaylov and Frank, 2018).”
• A Survey on Application of Knowledge Graph (2020, background): “Hao et al. [18] provide an end-to-end neural network model with cross-attention mechanism which considers various candidate answer aspects to represent the questions and their corresponding scores.”
• Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding (2021, background): “Often represented as large scale multi-relational graphs, Knowledge Graphs have seen wide application in a variety of domains, such as question answering (Yao and Van Durme, 2014; Hao et al., 2017), and natural language processing (Berant et al., 2013; Yu and Dredze, 2014), to name a few.”
• Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings (2020, background): “Methods like (Dai et al., 2016; Dong et al., 2015; Hao et al., 2017; Lukovnikov et al., 2017; Yin et al., 2016) utilize neural networks to learn a scoring functions to rank the candidate answers.”
• Towards Scalable and Reliable Capsule Networks for Challenging NLP Applications (2019, background): “Those forms are adopted to match answers from structured knowledge bases (Yao and Van Durme, 2014; Yih et al., 2015; Bor-des et al., 2015; Yin et al., 2016; Hao et al., 2017).”
• Unifying Knowledge Graph Learning and Recommendation: Towards a Better Understanding of User Preferences (2019, background): “Since it’s proposed, KG has attracted much attention in many research areas, ranging from recommendation [40], question answering [11], to information extraction [3].”
• A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network (2017, background): “These KBs are useful resources in many applications such as semantic searching and ranking (Kasneci et al., 2008; Schuhmacher and Ponzetto, 2014; Xiong et al., 2017), question answering (Zhang et al., 2016; Hao et al., 2017) and machine reading (Yang and Mitchell, 2017).”
• Neural Snowball for Few-Shot Relation Learning (2019, background): “, 2017), question answering (Hao et al., 2017) and recommender systems (Zhang et al.”
• Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization (2018, methodology): “Instead of using a geometric mean as in [57], we propose to use cross attention [21, 37] as the second-order term to increase the capacity.”
• Evaluating word embedding models: methods and experimental results (2019, methodology): “is a powerful tool widely used in modern natural language processing (NLP) tasks, including semantic analysis [1], information retrieval [2], dependency parsing [3], [4], [5], question answering [6], [7] and machine translation [6], [8], [9]. Learning a high quality representation is extremely impor”
• A Survey on Text Classification: From Shallow to Deep Learning (2020, background): “…attention mechanism [127] P, MAP, MRR TREC-QA, WikiQA, etc. Attention Inner-Attention mechanism [128] Accuracy SNLI cross-attention mechanism [129] F1 WebQuestion self-attention sentence embedding [130] Accuracy Yelp, Age sequence generation model [74] HL, M icro − F 1 RCV1-V2, AAPD deep…”
• Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering (2019, background): “…are not provided, researchers have trained such components by retrieving justifications from structured KBs (Cui et al., 2017; Bao et al., 2016; Zhang et al., 2016; Hao et al., 2017) such as ConceptNet (Speer et al., 2017), or from IR systems coupled with denoising components (Wang et al., 2019).”
• RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space (2018, background): “Knowledge graphs are potentially useful to a variety of applications such as question-answering (Hao et al., 2017), information retrieval (Xiong et al.”