Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers

By Anne Lauscher, Olga Majewska, Leonardo F. R. Ribeiro, Iryna Gurevych, Nikolai Rozanov, Goran Glavavs. 2020

93 citations recorded by the source corpus.

Following the major success of neural language models (LMs) such as BERT or GPT-2 on a variety of language understanding tasks, recent work focused on injecting (structured) knowledge from external resources into these models. While on the one hand, joint pre-training (i.e., training from scratch, adding objectives based on external knowledge to the primary LM objective) may be prohibitively computationally expensive, post-hoc fine-tuning on external knowledge, on the other hand, may lead to the catastrophic forgetting of distributional knowledge. In this work, we investigate models for complementing the distributional knowledge of BERT with conceptual knowledge from ConceptNet and its corresponding Open Mind Common Sense (OMCS) corpus, respectively, using adapter training. While overall results on the GLUE benchmark paint an inconclusive picture, a deeper analysis reveals that our adapter-based models substantially outperform BERT (up to 15-20 performance points) on inference tasks that require the type of conceptual knowledge explicitly present in ConceptNet and OMCS. We also open source all our experiments and relevant code under: https://github.com/wluper/retrograph.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

AdapterDrop: On the Efficiency of Adapters in Transformers (2020, background): “…transfer (Pfeiffer et al., 2020b, 2021b; Üstün et al., 2020; Vidoni et al., 2020; Ansell et al., 2021), community QA (Rücklé et al., 2020), and task composition for transfer learning (Stickland and Murray, 2019; Pfeiffer et al., 2021a; Lauscher et al., 2020; Wang et al., 2021; Poth et al., 2021).”
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, background): “, 2020) transfer learning (Stickland and Murray, 2019; Wang et al., 2020; Lauscher et al., 2020), and cross-lingual transfer (Pfeiffer et al.”
UNKs Everywhere: Adapting Multilingual Language Models to New Scripts (2020, background): “The largest drops are reported for resourcepoor target languages (Lauscher et al., 2020b), and (even more dramatically) for languages not covered at all during pretraining (Pfeiffer et al., 2020c).”
Retrieval-Free Knowledge-Grounded Dialogue Response Generation with Adapters (2021, background): “Recently, large pre-trained language models (LMs) have been shown to have the capability to carry implicit knowledge (Wang et al. 2020; Lauscher et al. 2020), which can be further applied to downstream classification tasks (Shwartz et al.”
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, methodology): “Another observation concerns substantially reduced cross-lingual and monolingual abilities of the models for resource-poor languages with smaller pretraining data (Wu and Dredze, 2020; Lauscher et al., 2020b).”
Relational World Knowledge Representation in Contextual Language Models: A Review (2021, background): “Compositional paths spanning multiple atoms of symbolic knowledge may also be lexicalized and input to an LM (Lauscher et al., 2020; Talmor et al., 2020a) in order to train LMs for soft com-Figure positional reasoning (Clark et al., 2020; Talmor et al., 2020b).”