Mimicking Word Embeddings using Subword RNNs

By Yuval Pinter, Robert Guthrie, Jacob Eisenstein. 2017

172 citations recorded by the source corpus.

Word embeddings improve generalization over lexical features by placing each word in a lower-dimensional space, using distributional information obtained from unlabeled data. However, the effectiveness of word embeddings for downstream NLP tasks is limited by out-of-vocabulary (OOV) words, for which embeddings do not exist. In this paper, we present MIMICK, an approach to generating OOV word embeddings compositionally, by learning a function from spellings to distributional embeddings. Unlike prior work, MIMICK does not require re-training on the original word embedding corpus; instead, learning is performed at the type level. Intrinsic and extrinsic evaluations demonstrate the power of this simple approach. On 23 languages, MIMICK improves performance over a word-based baseline for tagging part-of-speech and morphosyntactic attributes. It is competitive with (and complementary to) a supervised character-based model in low resource settings.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

BPE-Dropout: Simple and Effective Subword Regularization (2019, methodology): “Edit distance is commonly used to model misspellings (Brill and Moore, 2000; Ahmad and Kondrak, 2005; Pinter et al., 2017).”
Recent Trends in Deep Learning Based Natural Language Processing (2017, neutral): “[38] provided an interesting approach Fig.”
Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimicking (2019, methodology): “The former use subword information to infer a word’s meaning; this includes n -grams (Wieting et al. 2016; Bojanowski et al. 2017; Salle and Villavicencio 2018), morphemes (Lazari-dou et al. 2013; Luong, Socher, and Manning 2013) and characters (Pinter, Guthrie, and Eisenstein 2017).”
Dynamic Language Models for Continuously Evolving Content (2021, background): “In a separate line of studies, researchers break-down a word to more fine-grained units, including characters [2, 20, 28, 32, 49], character-based ngrams [7, 41, 45], and subwords (eg, wordpiece [46] and byte-pairencodings [22, 37]).”
Log-based Anomaly Detection Without Log Parsing (2021, methodology): “Besides, Log2Vec also adopts MIMICK [49], an approach of inducing word embedding from character-level features to handle OOV words to improve anomaly detection performance.”
Recent Trends in Deep Learning Based Natural Language Processing [Review Article] (2018, background): “[38] provided an interesting approach of training a character-based model to recreate pre-trained embeddings.”
Learning Semantic Representations for Novel Words: Leveraging Both Form and Context (2018, methodology): “The ﬁrst direction aims to obtain good embeddings for novel words by looking at their characters (Pinter, Guthrie, and Eisenstein 2017), morphemes (Lazari-dou et al. 2013; Luong, Socher, and Manning 2013; Cot-terell, Sch¨utze, and Eisner 2016) or n -grams (Wieting et al. 2016; Bojanowski et al.…”
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, background): “Speciﬁcally, Pinter et al. (2017) learn a model that is trained to mimic the embedding of a word given its spelling using a helper RNN model that is called whenever an unknown word appears during test time.”
Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model (2018, background): “…language models is the work of Bhatia, Guthrie, and Eisen-stein (2016), who similarly realize that placing priors on word embeddings is better than compositional construction, and Pinter, Guthrie, and Eisenstein (2017), who prove that the spelling of a word shares information with its embedding.”