StructGPT: A General Framework for Large Language Model to Reason over Structured Data

By Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, Ji-rong Wen. Conference on Empirical Methods in Natural Language Processing, 2023

580 citations recorded by the source corpus.

In this paper, we study how to improve the zero-shot reasoning ability of large language models~(LLMs) over structured data in a unified way. Inspired by the study on tool augmentation for LLMs, we develop an \emph{Iterative Reading-then-Reasoning~(IRR)} approach for solving question answering tasks based on structured data, called \textbf{StructGPT}. In our approach, we construct the specialized function to collect relevant evidence from structured data (\ie \emph{reading}), and let LLMs concentrate the reasoning task based on the collected information (\ie \emph{reasoning}). Specially, we propose an \emph{invoking-linearization-generation} procedure to support LLMs in reasoning on the structured data with the help of the external interfaces. By iterating this procedures with provided interfaces, our approach can gradually approach the target answer to a given query. Extensive experiments conducted on three types of structured data demonstrate the effectiveness of our approach, which can significantly boost the performance of ChatGPT and achieve comparable performance against the full-data supervised-tuning baselines. Our codes and data are publicly available at~\url{https://github.com/RUCAIBox/StructGPT}.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e0f27336698c84709bd60b6b7f4ce588cbae66bf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wayne Xin Zhao
2. Ji-rong Wen
3. Yupeng Hou
4. Hongyu Lu
5. Kun Zhou
6. Shirui Pan
7. Bowen Zheng
8. Yu Chen

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Ji-rong Wen
3. Wayne Xin Zhao
4. Xindong Wu
5. Kun Zhou
6. Shirui Pan
7. Jiawei Han
8. Heng Ji

Citations in the corpus

• Graph Retrieval-Augmented Generation: A Survey (2024, unknown): “For instance, Subgraph Retriever [181] adopts an alternating training paradigm, where the retriever’s parameters are fixed to use the graph data for training the generator.”
• Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph (2023, methodology): “Researchers (Li et al., 2023b; Xie et al., 2022; Baek et al., 2023b; Yang et al., 2023; Wang et al., 2023a; Jiang et al., 2023) have explored the usage of KGs as external knowledge sources to mitigate hallucination in LLMs.”
• Evaluating Object Hallucination in Large Vision-Language Models (2023, background): “, 2022), knowledge utilization (Jiang et al., 2023), and complex reasoning (Wei et al.”
• A Survey on Evaluation of Large Language Models (2023, unknown): “These include tasks such as mathematical reasoning [214, 224, 230] and structured data inference [79, 141].”
• Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation (2023, background): “Recently, large language models (LLMs) have gained significant popularity, with a wide range of applications spanning various domains of artificial intelligence [16], [28]–[30].”
• Think-on-Graph: Deep and Responsible Reasoning of Large Language Model with Knowledge Graph (2023, methodology): “Researchers have explored incorporating information from KGs Li et al. (2023b); Xie et al. (2022); Baek et al. (2023b); Yang et al. (2023); Wang et al. (2023a); Jiang et al. (2023a) and adapting prompts accordingly to mitigate hallucination in LLMs.”
• HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models (2023, background): “…in text summarization (Zhao et al., 2020), invoke interfaces of structured data ( e.g., knowledge graph, database) to obtain related evidence (Jiang et al., 2023; Lan et al., 2022), and train a token-level fact critic to recognize hallucination and rectify them in dialogue (Dziri et al.,…”
• Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, methodology): “…[42] 2021 KEPLER [40] 2021 DRAGON [44] 2022 HKLM [238] 2023 Synergized Reasoning LARK [45] 2023 Siyuan et al. [46] 2023 KSL [239] 2023 StructGPT [237] 2023 Think-on-graph [240] 2023 To better guide LLMs reason through KGs, OreoLM [177] proposes a Knowledge Interaction Layer (KIL) which is…”
• Language is All a Graph Needs (2023, methodology): “…employing LLM’s zero-shot explanations for data augmentation to obtain advanced graph features (He et al., 2023), generating prompts and benchmarks for graph construction, evaluation, biology and structural reasoning (Han et al., 2023b; Jiang et al., 2023; Qian et al., 2023; Guo et al., 2023).”
• Mind2Web: Towards a Generalist Agent for the Web (2023, background): “Meanwhile, LLMs have been shown to excel at grounded language understanding in complex environments with good generalizability and sample efficiency [2, 13, 17, 33].”
• G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering (2024, unknown): “Noteworthy applications include fundamental graph reasoning [52, 3, 56], node classification [8, 11, 39, 5, 50, 4, 33], graph classification/regression [32, 55], and leveraging LLMs for knowledge graph-related tasks [41, 14, 29].”
• Large Language Models on Graphs: A Comprehensive Survey (2023, background): “Another approach is to iteratively retrieve and reason on the subgraphs from KG [130], [132], simulating a dynamic searching process.”
• Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “StructGPT [298] extracts relevant information from knowledge graph, table, or database, to form prompt and generate answers through LLMs. cTBLS [324] retrieves relevant tables through dense retrieval, then for each query, it forms prompt with ranked tables for answer generation.”
• MARS: Multi-hop Adaptive Retrieval and SPARQL Generation for KGQA (2026, unknown): “Relevant evidence collected from structured data with an iterative reading-then-reasoning approach in zero-shot and few-shot scenarios achieved promising results with LLMs [20].”
• ArchRAG: Attributed Community-based Hierarchical Retrieval-Augmented Generation (2025, unknown): “These include using LLMs for KG construction [86], and employing KG to enhance the LLM reasoning [27, 29, 41, 45, 58, 67, 69].”