Learning Semantic Representations for Novel Words: Leveraging Both Form and Context

By Timo Schick, Hinrich Schütze. 2018

36 citations recorded by the source corpus.

Word embeddings are a key component of high-performing natural language processing (NLP) systems, but it remains a challenge to learn good representations for novel words on the fly, i.e., for words that did not occur in the training data. The general problem setting is that word embeddings are induced on an unlabeled training corpus and then a model is trained that embeds novel words into this induced embedding space. Currently, two approaches for learning embeddings of novel words exist: (i) learning an embedding from the novel word’s surface-form (e.g., subword n-grams) and (ii) learning an embedding from the context in which it occurs. In this paper, we propose an architecture that leverages both sources of information – surface-form and context – and show that it results in large increases in embedding quality. Our architecture obtains state-of-the-art results on the Definitional Nonce and Contextual Rare Words datasets. As input, we only require an embedding set and an unlabeled corpus for training our architecture to produce embeddings appropriate for the induced embedding space. Thus, our model can easily be integrated into any existing NLP system and enhance its capability to handle novel words.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, methodology): “MultiRC Deviating from the hyperparameters used by Schick and Sch¨utze (2019), we use a maximum sequence length of 512 tokens for MultiRC both during training and inference because we found many passages to be much longer than 256 tokens.”
Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimicking (2019, methodology): “The form-context model and its Attentive Mimicking variant (Schick and Sch¨utze 2019a) achieve a new state-of-the-art for high-quality representations of rare words.”
Learning to Learn Words from Visual Scenes (2019, background): “Out-of-vocabulary words: This paper is related but different to models of out-of-vocabulary words (OOV) [45,25,18,23,44,43,19,45].”