K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

By Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, Ming Zhou. 2020

629 citations recorded by the source corpus.

We study the problem of injecting knowledge into large pre-trained models like BERT and RoBERTa. Existing methods typically update the original parameters of pre-trained models when injecting knowledge. However, when multiple kinds of knowledge are injected, they may suffer from catastrophic forgetting. To address this, we propose K-Adapter, which remains the original parameters of the pre-trained model fixed and supports continual knowledge infusion. Taking RoBERTa as the pre-trained model, K-Adapter has a neural adapter for each kind of infused knowledge, like a plug-in connected to RoBERTa. There is no information flow between different adapters, thus different adapters are efficiently trained in a distributed way. We inject two kinds of knowledge, including factual knowledge obtained from automatically aligned text-triplets on Wikipedia and Wikidata, and linguistic knowledge obtained from dependency parsing. Results on three knowledge-driven tasks (total six datasets) including relation classification, entity typing and question answering demonstrate that each adapter improves the performance, and the combination of both adapters brings further improvements. Probing experiments further indicate that K-Adapter captures richer factual and commonsense knowledge than RoBERTa.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

Entities as Experts: Sparse Memory Access with Entity Supervision (2020, background): “…Yang et al., 2016; Logan et al., 2019; Zhang et al., 2019; Levine et al., 2019; Xiong et al., 2019a; Peters et al., 2019; Poerner et al., 2019; Wang et al., 2020 Memory Augmented Neural Networks Our entity memory layer is closely tied to memory-based neural layers (Weston et al., 2014;…”
AdapterDrop: On the Efficiency of Adapters in Transformers (2020, background): “, 2020), and task composition for transfer learning (Stickland and Murray, 2019; Pfeiffer et al., 2021a; Lauscher et al., 2020; Wang et al., 2021; Poth et al., 2021).”
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, methodology): “, 2020) transfer learning (Stickland and Murray, 2019; Wang et al., 2020; Lauscher et al., 2020), and cross-lingual transfer (Pfeiffer et al.”
Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs (2021, methodology): “We also construct (4) a sequence prediction task using data from Wikidata5m, which is a relational knowledge base with over 20 million triplets (Wang et al., 2021b).”
DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, background): “Prompts capture task-specific knowledge with much smaller additional parameters, than its competitors, such as Adapter [57, 42] and LoRA [16].”
Learning to Prompt for Continual Learning (2021, background): “Prompts capture task-specific knowledge with much smaller additional parameters, than its competitors, such as Adapter [43, 58] and LoRA [18].”
GAN Memory with No Forgetting (2020, background): “However, most existing generative replay methods either deliver blurry generated samples [10, 40] or only work well on simple datasets [40, 78, 40] like MNIST; besides, they often don’t scale well to practical situations with high resolution [60] or a long sequence [86], sometimes even with negative”
Pre-trained models for natural language processing: A survey (2020, background): “To address this, K-Adapter [191] injects multiple kinds of knowledge by training different adapters independently for different pre-training tasks, which allows continual knowledge infusion.”
ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation (2021, methodology): “K-Adapter[31] utilized adapters trained on different knowledge sources with extra annotations to distinguish where the knowledge comes from.”
Prompt Injection: Parameterization of Fixed Inputs (2022, neutral): “Updating or adding a relatively small number of parameters [11, 31] may be a potential avenue for addressing the problems.”
Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers (2020, background): “While (Wang et al., 2020) inject factual knowledge from Wikidata (Vrandečić and Krötzsch, 2014) into BERT, in this work, we investigate two resources that are commonly assumed to contain generalpurpose and common sense knowledge:1 ConceptNet (Liu and Singh, 2004; Speer et al.”