Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimicking

By Timo Schick, Hinrich Schütze. 2019

107 citations recorded by the source corpus.

Pretraining deep neural network architectures with a language modeling objective has brought large improvements for many natural language processing tasks. Exemplified by BERT, a recently proposed such architecture, we demonstrate that despite being trained on huge amounts of data, deep language models still struggle to understand rare words. To fix this problem, we adapt Attentive Mimicking, a method that was designed to explicitly learn embeddings for rare words, to deep language models. In order to make this possible, we introduce one-token approximation, a procedure that enables us to use Attentive Mimicking even when the underlying language model uses subword-based tokenization, i.e., it does not assign embeddings to all words. To evaluate our method, we create a novel dataset that tests the ability of language models to capture semantic properties of words without any task-specific fine-tuning. Using this dataset, we show that adding our adapted version of Attentive Mimicking to BERT does substantially improve its understanding of rare words.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, methodology): “With the rise of pretrained language models such as GPT (Radford et al., 2018), BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), the idea of providing task descriptions has become feasible for neural architectures: We can simply append such descriptions in natural language to an input and…”
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, methodology): “As ﬁnding ways to reformulate tasks as cloze questions that are understood well by LMs is difﬁ-cult (Jiang et al., 2019), Schick and Sch¨utze (2020a) propose P ET , a method that uses knowledge distillation (Hinton et al., 2015) to easily combine several reformulations.”
Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification (2020, methodology): “As recently shown by Schick and Schütze (2020b), pretrained language models tend to understand frequent words much better than rare words, so all other things being equal, a frequent word should be preferred over a rare word as verbalization; that is, this bias towards frequent words is indeed…”
What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge (2019, background): “There have been recent studies into the types of relational knowledge contained in large-scale knowledge models (Schick and Schütze, 2020; Petroni et al., 2019; Jiang et al., 2019), which also probe models using structured knowledge sources.”
Taking Notes on the Fly Helps BERT Pre-training (2020, unknown): “A large proportion of words appear only very few times and the embeddings of those (rare) words are usually poorly optimized (Bahdanau et al., 2017; Gong et al., 2018; Khassanov et al., 2019; Schick & Sch¨utze, 2020).”
Learning to Learn Words from Visual Scenes (2019, background): “We compare to the most competitive approach [45], which reduces to regular BERT in our setting, as a baseline.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “Unlike these probes, WDLMPro and WNLaMPro focus on understanding how the pretrained models understand the words.”
Few-Shot Text Generation with Pattern-Exploiting Training (2020, methodology): “For the learning rate, we tried values of α · 10 − 5 with α ∈ { 1 , 10 , 50 } as Schick and Schütze (2020a) use α = 1 and Zhang et al. (2020) use α = 50 ; we found α = 10 to perform best for all models.”
Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-Modal Knowledge Transfer (2022, background): “Though yielding good performance on various NLU downstream tasks, these pre-training objectives suffer from a lack of out-of-domain knowledge that is not explicitly present in the pre-training corpus (Gururangan et al., 2020a; Petroni et al., 2021; Schick and Schütze, 2020).”