UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models

By Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang and 13 more. 2022

362 citations recorded by the source corpus.

Structured knowledge grounding (SKG) leverages structured knowledge to complete user requests, such as semantic parsing over databases and question answering over knowledge bases. Since the inputs and outputs of SKG tasks are heterogeneous, they have been studied separately by different communities, which limits systematic and compatible research on SKG. In this paper, we overcome this limitation by proposing the UnifiedSKG framework, which unifies 21 SKG tasks into a text-to-text format, aiming to promote systematic SKG research, instead of being exclusive to a single task, domain, or dataset. We use UnifiedSKG to benchmark T5 with different sizes and show that T5, with simple modifications when necessary, achieves state-of-the-art performance on almost all of the 21 tasks. We further demonstrate that multi-task prefix-tuning improves the performance on most tasks, largely improving the overall performance. UnifiedSKG also facilitates the investigation of zero-shot and few-shot learning, and we show that T0, GPT-3, and Codex struggle in zero-shot and few-shot learning for SKG. We also use UnifiedSKG to conduct a series of controlled experiments on structured knowledge encoding variants across SKG tasks. UnifiedSKG is easily extensible to more tasks, and it is open-sourced at https://github.com/hkunlp/unifiedskg.

Subfields: Artificial Intelligence, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “UnifiedSKG (Xie et al., 2022) is a collection of 21 tasks related to Structured Knowledge Grounding with heterogeneous inputs such as databases, dialogue states, SQL queries, etc.”
QLoRA: Efficient Finetuning of Quantized LLMs (2023, neutral): “…datasets include MetaICL [40], MetaTuning [74], InstructGPT [43], FLAN [63, 12], PromptSource [3], Super-NaturalInstructions [62, 51], Self-instruct [60], UnnaturalInstructions [26], OPT-IML [29], UnifiedSKG[68], OIG/Chip2 [32], Alpaca [56], Vicuna [10], Koala [20], and Self-instruct-GPT-4 [45].”
Instruction Tuning with GPT-4 (2023, methodology): “Instruction-tuning of OPT models is considered in (Iyer et al., 2022), where a larger and more comprehensive benchmark OPT-IML Bench is employed, covering FLAN (Wei et al., 2021), Super-NaturalInstructions (Wang et al., 2022b), and UnifiedSKG (Xie et al., 2022).”
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, background): “More recent studies show promise that large-scale multi-task learning can enable strong generalization to similar tasks via unified encoding (Khashabi et al., 2020; Xie et al., 2022) or better finetuning results on downstream tasks (McCann et al.”
Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, methodology): “Invasive meth-ods, which are built on a strong assumption that the inner structure (e.g., self-attention and feed-forward layers) of the PLM can be modiﬁed, includes Preﬁx-Tuning (Li and Liang, 2021), Bit-ﬁt (Ben Zaken et al., 2021), Child-Tuning (Xu et al., 2021), P-Tuning v2 (Liu et al., 2021b), L”
Retrieval-Augmented Multimodal Language Modeling (2023, background): “…et al., 2018; Khandelwal et al., 2019; Karpukhin et al., 2020; Guu et al., 2020; Lewis et al., 2020b;a; Yasunaga et al., 2022b; Borgeaud et al., 2022; Shi et al., 2023) or a structured knowledge base (Zhang et al., 2019; Agarwal et al., 2021; Xie et al., 2022; Yasunaga et al., 2021; 2022a).”
Binding Language Models in Symbolic Languages (2022, methodology): “…we refer readers interested in these baselines to their respective papers for details (Liu et al., 2021; Jiang et al., 2022; Zhou et al., 2022; Xie et al., 2022; Raffel et al., 2020; Ou & Liu, 2022), and we present a complete list and evaluation results of more previous systems on these…”
Prompt Injection: Parameterization of Fixed Inputs (2022, neutral): “The domain-adaptation setup introduces many generalization challenges, such as non-explicit column names and domain-specific phrases [10], and recent works concatenate the natural language query with the serialized database schema as the input to address the problem [26, 7, 36].”
Benchmarking Generalization via In-Context Instructions on 1, 600+ Language Tasks (2022, background): “More recent studies show tremendous promise in generalization within the set of observed tasks through multi-task training and uniﬁed encoding (Khashabi et al., 2020; Aghajanyan et al., 2021; Xie et al., 2022; Aribandi et al., 2022).”
Unifying Language Learning Paradigms (2022, methodology): “We use several component tasks from UniﬁedSKG (Xie et al., 2022), namely WikiTQ (Pasupat & Liang, 2015), CompWQ (Talmor & Be-rant, 2018), FetaQA (Nan et al., 2021), HybridQA (Chen et al., 2020), WikiSQL (Zhong et al., 2017), TabFat (Chen et al., 2019), Feverous (Aly et al., 2021), SQA (Iyyer et al.,”
Think-on-Graph: Deep and Responsible Reasoning of Large Language Model with Knowledge Graph (2023, background): “Researchers have explored incorporating information from KGs Li et al. (2023b); Xie et al. (2022); Baek et al. (2023b); Yang et al. (2023); Wang et al. (2023a); Jiang et al. (2023a) and adapting prompts accordingly to mitigate hallucination in LLMs.”
AlignScore: Evaluating Factual Consistency with A Unified Alignment Function (2023, background): “Recent studies also present task unification as an effective approach to improve model performance and generalizability in multimodal tasks (Xie et al., 2022; Zhang et al., 2021; Wang et al., 2022).”
L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models (2023, methodology): “This is also consistent with previous work on L2C (Xie et al., 2022; Ni et al., 2023b; Shi et al., 2022).”
A Survey of Large Language Models (2023, neutral): “1M ExMix [196] Nov-2021 11 107 18M UnifiedSKG [197] Jan-2022 6 21 812K Super Nat.”