Knowledge Graph Prompting for Multi-Document Question Answering

By Yu Wang, Nedim Lipka, Ryan A. Rossi, Alexa F. Siu, Ruiyi Zhang, Tyler Derr. AAAI Conference on Artificial Intelligence, 2023

329 citations recorded by the source corpus.

The `pre-train, prompt, predict' paradigm of large language models (LLMs) has achieved remarkable success in open-domain question answering (OD-QA). However, few works explore this paradigm in multi-document question answering (MD-QA), a task demanding a thorough understanding of the logical associations among the contents and structures of documents. To fill this crucial gap, we propose a Knowledge Graph Prompting (KGP) method to formulate the right context in prompting LLMs for MD-QA, which consists of a graph construction module and a graph traversal module. For graph construction, we create a knowledge graph (KG) over multiple documents with nodes symbolizing passages or document structures (e.g., pages/tables), and edges denoting the semantic/lexical similarity between passages or document structural relations. For graph traversal, we design an LLM-based graph traversal agent that navigates across nodes and gathers supporting passages assisting LLMs in MD-QA. The constructed graph serves as the global ruler that regulates the transitional space among passages and reduces retrieval latency. Concurrently, the graph traversal agent acts as a local navigator that gathers pertinent context to progressively approach the question and guarantee retrieval quality. Extensive experiments underscore the efficacy of KGP for MD-QA, signifying the potential of leveraging graphs in enhancing the prompt design and retrieval augmented generation for LLMs. Our code: https://github.com/YuWVandy/KG-LLM-MDQA.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/629f44f5fb78ec390ef66633dc627f1d04f3eb85

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Heng Ji
2. Jiawei Han
3. Meng Jiang
4. Bowen Jin
5. Chi Han
6. Gang Liu
7. Haofen Wang
8. Cynthia Rudin

Citations in the corpus

• From Local to Global: A Graph RAG Approach to Query-Focused Summarization (2024, methodology): “…subgraphs (SURGE, Kang et al., 2023), where retrieved event-plot sub-graphs are serialized using narrative templates (FABULA, Ranade and Joshi, 2023), and where the system supports both creation and traversal of text-relationship graphs for multi-hop question answering (Wang et al., 2023b).”
• Retrieval-Augmented Generation for Large Language Models: A Survey (2023, methodology): “It has also introduced various methods to optimize the retrieval process [ILIN, 2023].”
• Large Language Models on Graphs: A Comprehensive Survey (2023, background): “Many graph-based concepts, such as shortest paths, particular sub-graphs, and flow networks, have strong connections with real-world applications [133]– [135], [193].”
• Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, unknown): “KD-CoT [247] iteratively retrieves facts from KGs and produces faithful reasoning traces, which guide LLMs to generate answers.”
• Graph Retrieval-Augmented Generation: A Survey (2024, unknown): “…between the documents, Munikoti et al. [113] propose to construct a heterogeneous document graph capturing multiple document-level relations, including co-citation, co-topic, co-venue, etc. Li et al. [87] and Wang et al. [160] establish relationship between passages according to shared keywords.”
• A Matryoshka Hierarchical RAG for Efficient Multi-Hop Question Answering (2026, unknown): “KGP [39] moves further away from entities by building a graph of passages with edges based on semantic and structural relationships traversed by an LLM-based agent.”
• ArchRAG: Attributed Community-based Hierarchical Retrieval-Augmented Generation (2025, unknown): “These include using LLMs for KG construction [86], and employing KG to enhance the LLM reasoning [27, 29, 41, 45, 58, 67, 69].”
• Knowledge Graph-Guided Retrieval Augmented Generation (2025, unknown): “For example, KGP (Wang et al., 2024a) constructs a document KG consisting of page and passage nodes, and links passage nodes with TF-IDF.”