Learning to Compute Word Embeddings On the Fly

By Dzmitry Bahdanau, Tom Bosc, Stanislaw Jastrzebski, Edward Grefenstette, Pascal Vincent, Yoshua Bengio. 2017

86 citations recorded by the source corpus.

Words in natural language follow a Zipfian distribution whereby some words are frequent but most are rare. Learning representations for words in the ``long tail'' of this distribution requires enormous amounts of data. Representations of rare words trained directly on end tasks are usually poor, requiring us to pre-train embeddings on external data, or treat all rare words as out-of-vocabulary words with a unique representation. We provide a method for predicting embeddings of rare words on the fly from small amounts of auxiliary data with a network trained end-to-end for the downstream task. We show that this improves results against baselines where embeddings are trained on the end task for reading comprehension, recognizing textual entailment and language modeling.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Dynamic Language Models for Continuously Evolving Content (2021, background): “Most works adopt a simple pooling approach, eg , summation [15], mean pooling [3, 19] to aggregate the embeddings of the contextual words as the OOV word embeddings, while Hu et al. [17] propose an attention-based hierarchical context encoder to encode and aggregate both context and subword…”
Dynamic Integration of Background Knowledge in Neural NLU Systems (2017, methodology): “Bahdanau et al. (2017) and Long et al. (2017) create word embeddings on-the-fly by reading word definitions prior to processing the task at hand.”
Latent Relation Language Models (2019, background): “, 2019), definitions (Bahdanau et al., 2017), or keywords (Kiddon et al.”
Taking Notes on the Fly Helps BERT Pre-training (2020, unknown): “A large proportion of words appear only very few times and the embeddings of those (rare) words are usually poorly optimized (Bahdanau et al., 2017; Gong et al., 2018; Khassanov et al., 2019; Schick & Sch¨utze, 2020).”
Making Neural Machine Reading Comprehension Faster (2019, methodology): “Most of the papers that approach to solve SQuAD problem have utilized RNN based models [3, 5, 7, 9, 13, 14, 16, 17, 20, 22–24, 26, 29–31, 34–39, 41, 43] and they follow a similar chain of processes.”
Relational Memory-Augmented Language Models (2022, unknown): “, 2018b), and entity descriptions (Bahdanau et al., 2017).”