High-risk learning: acquiring new word vectors from tiny data

By Aurélie Herbelot, Marco Baroni. 2017

84 citations recorded by the source corpus.

Distributional semantics models are known to struggle with small data. It is generally accepted that in order to learn ‘a good vector’ for a word, a model must have sufficient examples of its usage. This contradicts the fact that humans can guess the meaning of a word from a few occurrences only. In this paper, we show that a neural language model such as Word2Vec only necessitates minor modifications to its standard architecture to learn new terms from tiny data, using background knowledge from a previously learnt semantic space. We test our model on word definitions and on a nonce task involving 2-6 sentences’ worth of context, showing a large increase in performance over state-of-the-art models on the definitional task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Recent Trends in Deep Learning Based Natural Language Processing (2017, background): “[37] provided OOV handling on-the-fly by initializing the unknown words as the sum of the context words and refining these words with a high learning rate.”
Generalizing from a Few Examples (2019, neutral): “Example applications include parsing [64], translation [65], sentence completion (which fills in the blanks using a word chosen from a provided set) [97, 138], sentiment classification from short reviews [150, 157], user intent classification for dialog systems [157], criminal charge prediction [61]”
Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimicking (2019, background): “On the other hand, context-based approaches take a look at the words surrounding a given rare word to obtain a representation for it (e.g., Herbelot and Baroni 2017; Khodak et al. 2018).”
Dynamic Language Models for Continuously Evolving Content (2021, background): “The third line of research attempts to explicitly generate OOV word embeddings “on the fly” from context such as the definitions of the OOV word in a dictionary [3] and example sentences that contain the OOV word [15, 17, 19, 26].”
Memory, Show the Way: Memory Based Few Shot Word Representation Learning (2018, methodology): “Directly summing the context as SUM does or taking risk to enlarge the window size as N2V may lead to over-ﬁtting.”
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2021, background): “While most work on few-shot focuses on computer vision, there is an increasing attention in NLP recently (Han et al., 2018; Rios and Kavuluru, 2018; Hu et al., 2018; Herbelot and Baroni, 2017).”
A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors (2018, background): “” Stop-word removal can help to ameliorate this (Lazaridou et al., 2017; Herbelot and Baroni, 2017), but does not deal with the fact that content-words also have significant components in the same direction as these deleted words.”
Recent Trends in Deep Learning Based Natural Language Processing [Review Article] (2018, background): “Herbelot and Baroni [37] provided OOV handling on-the-fly by”
Learning Semantic Representations for Novel Words: Leveraging Both Form and Context (2018, methodology): “Ourapproach is able to generate embeddings for OOV words even from only a single observation with high accuracy in many cases and outperforms previous work on the Deﬁnitional Nonce dataset (Herbelot and Baroni 2017) and the Contextual Rare Words dataset (Khodak et al. 2018).”
Learning to Learn Words from Visual Scenes (2019, background): “Out-of-vocabulary words: This paper is related but different to models of out-of-vocabulary words (OOV) [45,25,18,23,44,43,19,45].”
Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks (2018, background): “This is arguably more of an instance of the problem of quickly learning meaningful new-word embeddings (Herbelot and Baroni, 2017; Lampinen and McClelland, 2017), than strictly a failure of compositionality.”