A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors

By Mikhail Khodak, Nikunj Saunshi, Yingyu Liang, Tengyu Ma, Brandon M Stewart, Sanjeev Arora. 2018

109 citations recorded by the source corpus.

Motivations like domain adaptation, transfer learning, and feature learning have fueled interest in inducing embeddings for rare or unseen words, n-grams, synsets, and other textual features. This paper introduces a la carte embedding, a simple and general alternative to the usual word2vec-based approaches for building such representations that is based upon recent theoretical results for GloVe-like embeddings. Our method relies mainly on a linear transformation that is efficiently learnable using pretrained word vectors and linear regression. This transform is applicable on the fly in the future when a new text feature or rare word is encountered, even if only a single usage example is available. We introduce a new dataset showing how the a la carte method requires fewer examples of words in context to learn high-quality embeddings and we obtain state-of-the-art results on a nonce task and some unsupervised document classification tasks.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hinrich Schütze
2. Timo Schick
3. Qiang Liu
4. Mohit Bansal
5. Chenguang Zhu
6. Heng Ji
7. Weizhu Chen
8. Dilin Wang

Citations in the corpus

• Improving Neural Language Modeling via Adversarial Training (2019, methodology): “…method is that it provides an effective mechanism to encourage diversity of word embedding vectors, which is widely observed to yield better generalization performance in neural language models (e.g., Mu et al., 2018; Gao et al., 2019; Liu et al., 2018b; Cogswell et al., 2016; Khodak et al., 2018).”
• Continual Learning for Sentence Representations Using Conceptors (2019, background): “Building upon SIF, recent studies have proposed further improved sentence encoders (Khodak et al., 2018; Pagliardini et al., 2018; Yang et al., 2018).”
• Dynamic Language Models for Continuously Evolving Content (2021, background): “The third line of research attempts to explicitly generate OOV word embeddings “on the fly” from context such as the definitions of the OOV word in a dictionary [3] and example sentences that contain the OOV word [15, 17, 19, 26].”
• Zero-training Sentence Embedding via Orthogonal Basis (2018, methodology): “It also compares favorably with most of parametric models, including à la carte(Khodak et al., 2018), FastSent(Hill et al., 2016), InferSent, QT, Sent2Vec, SkipThought-LN (with layer normalization) (Kiros et al., 2015), SDAE(Hill et al., 2016), and STN (Subramanian et al., 2018).”
• Unsupervised Post-processing of Word Vectors via Conceptor Negation (2018, background): “For this reason, Khodak et al. (2018) argued that ABTT is liable either to not remove enough noise or to cause too much information loss.”
• Learning to Learn Words from Visual Scenes (2019, background): “Out-of-vocabulary words: This paper is related but different to models of out-of-vocabulary words (OOV) [45,25,18,23,44,43,19,45].”
• Learning Semantic Representations for Novel Words: Leveraging Both Form and Context (2018, methodology): “Very recently, Khodak et al. (2018) introduced the A La Carte embedding method that, similar to the summation model by Lazaridou, Marelli, and Baroni (2017), averages over all context words.”