Differentiable Reasoning over a Virtual Knowledge Base

By Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Ruslan Salakhutdinov, William W. Cohen. 2020

91 citations recorded by the source corpus.

We consider the task of answering complex multi-hop questions using a corpus as a virtual knowledge base (KB). In particular, we describe a neural module, DrKIT, that traverses textual data like a virtual KB, softly following paths of relations between mentions of entities in the corpus. At each step the operation uses a combination of sparse-matrix TFIDF indices and maximum inner product search (MIPS) on a special index of contextual representations. This module is differentiable, so the full system can be trained completely end-to-end using gradient based methods, starting from natural language inputs. We also describe a pretraining scheme for the index mention encoder by generating hard negative examples using existing knowledge bases. We show that DrKIT improves accuracy by 9 points on 3-hop questions in the MetaQA dataset, cutting the gap between text-based and KB-based state-of-the-art by 70%. DrKIT is also very efficient, processing upto 10x more queries per second than existing state-of-the-art QA systems.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, background): “…Retrieval (DR) using learned distributed representations is a promising direction to overcome this sparse retrieval bottleneck [9, 10, 11, 12, 13, 14, 15]: The representation space is fully learnable and can leverage the strength of pretraining, while the retrieval operation is sufﬁciently…”
SELFEXPLAIN: A Self-Explaining Architecture for Neural Text Classifiers (2021, methodology): “Differentiable approaches through Maximum Inner Product Search (MIPS) has been shown to be effective in QuestionAnswering settings (Guu et al., 2020; Dhingra et al., 2020) to leverage retrieved knowledge for reasoning 3.”
Adaptive Information Seeking for Open-Domain Question Answering (2021, methodology): “For example, some works employ a single retrieval function multiple times (Das et al., 2019a; Qi et al., 2019; Xiong et al., 2021), and the other works use a pre-deﬁned sequence of retrieval functions (Asai et al., 2020; Dhingra et al., 2020).”
Modifying Memories in Transformer Models (2020, background): “…there have been attempts to develop novel Transformer models and/or training procedures that aim to leverage both available high-quality KBs and large corpora of (unstructured) text (Dhingra et al., 2019; Guu et al., 2020; Lewis et al., 2020), further broadening the scope of factual knowledge.”
Pre-Trained Models: Past, Present and Future (2021, background): “Another kind of methods (Guo et al., 2019; Lee et al., 2019; Beltagy et al., 2020; Ainslie et al., 2020; Zaheer et al., 2020) combine global and local attention mechanisms, and then use global nodes to gather tokens in a sequence.”
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval (2020, methodology): “…that do not release the end-to-end code, we estimate the running time based on the number of BERT cross-attention forward passes (the same estimation strategy used by Dhingra et al. (2020)), and ignore the overhead caused by additional processing such as TF-IDF or linking graph construction.”
Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge (2020, background): “…has also been extended with works looking at ways of incorporating raw text and symbolic KGs into a shared embedding space (Riedel et al., 2013; Verga et al., 2016), to be jointly reasoned over (Sun et al., 2018, 2019a), or to treat text as a replacement for a knowledge base (Dhingra et al., 2019).”
Differentiable Open-Ended Commonsense Reasoning (2020, methodology): “6% over DrKIT (Dhingra et al., 2020) (a strong method for entity-centric multi-hop reasoning).”
Reasoning Over Virtual Knowledge Bases With Open Predicate Relations (2021, methodology): “, 2018) and DrKIT (Dhingra et al., 2020) build mention-level indexes using a pretrained encoder.”
A Review on Language Models as Knowledge Bases (2022, background): “RAG uses vector indices of Wikipedia to access latent information during inference on knowledge-intensive tasks (Lewis et al., 2021) Introduced by (Dhingra et al., 2020), DrKIT includes a virtual knowledge base (VKB), which is a "soft knowledge base" that is used in conjunction with neural methods…”
Relation-Guided Pre-Training for Open-Domain Question Answering (2021, unknown): “Recent studies (Lee et al., 2019; Karpukhin et al., 2020; Dhingra et al., 2020) use BERT-like pretrained language model to encode the question and passages independently into dense representations, and use maximum inner product search (MIPS) algorithms (Shrivastava and Li, 2014) to efﬁciently…”
Retrieve, Rerank, Read, then Iterate: Answering Open-Domain Questions of Arbitrary Complexity from Text (2020, background): “…systems, including making use of hyper-links between Wikipedia articles (Nie et al., 2019; Feldman and El-Yaniv, 2019; Zhao et al., 2019; Asai et al., 2020; Dhingra et al., 2020; Zhao et al., 2019) and iterative retrieval (Talmor and Berant, 2018; Das et al., 2019; Qi et al., 2019).”
Answering Any-hop Open-domain Questions with Iterative Document Reranking (2020, methodology): “BERT Reranker [5], DrKIT [8] and Transformer-XH [43] employ the entities in the question and the retrieved documents to link additional documents, which expands the one-shot retrieval results and improves the evidence coverage.”
Language Models are Open Knowledge Graphs (2020, background): “Studies have leveraged the pre-trained LMs as virtual KGs, and show reasonable performance in QA tasks (Dhingra et al., 2020; Guu et al., 2020), and language modeling (Khandelwal et al., 2019).”
SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval (2020, background): “Dhingra et al. (2020) incorporates entity vectors and multi-hop reasoning to teach systems to answer more complex questions.”
DDRQA: Dynamic Document Reranking for Open-domain Multi-hop Question Answering (2020, methodology): “(Feldman and El-Yaniv, 2019) 31.1 40.4 – 30.6 40.2 GoldEn Retriever (Qi et al., 2019) – 49.7 – 37.9 48.5 DrKIT (Dhingra et al., 2020) 37.9 48.6 – 42.1 51.7 Transformer-XH (Zhao et al., 2020) 54.0 7 points improvement on Paragraph EM over previous best model (Asai et al., 2020), demonstrating the…”
HopRetriever: Retrieve Hops over Wikipedia to Answer Complex Questions (2020, background): “In contrast, some others (Dhingra et al. 2020; Ding et al. 2019) focus on mentioned entities and try to traverse textual data like a virtual structured Knowledge Base (KB).”
Neural, symbolic and neural-symbolic reasoning on knowledge graphs (2021, background): “Other works such as [20], [32], [101] are similar works for multi-hop KGQA.”
F1 Is Not Enough! Models and Evaluation towards User-Centered Explainable Question Answering (2020, background): “, 2020), virtual knowledge bases (Dhingra et al., 2020) or transformer models (Zhao et al.”
Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering (2021, background): “After [3], lots of works have been released [27], [29], [32], [33], [36], [72], [73], [74].”