FRAGE: Frequency-Agnostic Word Representation

By Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, Tie-Yan Liu. 2018

147 citations recorded by the source corpus.

Continuous word representation (aka word embedding) is a basic building block in many neural network-based models used in natural language processing tasks. Although it is widely accepted that words with similar semantics should be close to each other in the embedding space, we find that word embeddings learned in several tasks are biased towards word frequency: the embeddings of high-frequency and low-frequency words lie in different subregions of the embedding space, and the embedding of a rare word and a popular word can be far from each other even if they are semantically similar. This makes learned word embeddings ineffective, especially for rare words, and consequently limits the performance of these neural network models. In this paper, we develop a neat, simple yet effective way to learn \emph{FRequency-AGnostic word Embedding} (FRAGE) using adversarial training. We conducted comprehensive studies on ten datasets across four natural language processing tasks, including word similarity, language modeling, machine translation and text classification. Results show that with FRAGE, we achieve higher performance than the baselines in all tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

BPE-Dropout: Simple and Effective Subword Regularization (2019, background): “This also relates to the study by Gong et al. (2018).”
AutoML: A Survey of the State-of-the-Art (2019, background): “, GPT-2 [148], FRAGE AWD-LSTM-Mos [4],”
Improving Neural Language Modeling via Adversarial Training (2019, background): “, 2016), and frequency agnostic training (Gong et al., 2018), etc.”
Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling (2019, background): “General language modeling Also related are the recent papers proposing modiﬁcations to the AWD-LSTM that improve performance on Wikitext-2 (Gong et al., 2018; Yang et al., 2018; Krause et al., 2018).”
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (2021, background): “As a result, models are brittle to rare words (Gong et al., 2018) and perturbations, both natural and adversarial (Belinkov and Bisk, 2018; Pruthi et al., 2019; Sun et al., 2020).”
Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code (2020, background): “and thus stay close to their initialized values [35].”
Double-Hard Debias: Tailoring Word Embeddings for Gender Bias Mitigation (2020, background): “Recent works (Mu and Viswanath, 2018; Gong et al., 2018) show that word frequency in a training”
Taking Notes on the Fly Helps BERT Pre-training (2020, unknown): “A large proportion of words appear only very few times and the embeddings of those (rare) words are usually poorly optimized (Bahdanau et al., 2017; Gong et al., 2018; Khassanov et al., 2019; Schick & Sch¨utze, 2020).”
A Simple Recurrent Unit with Reduced Tensor Product Representations (2018, methodology): “…entailment tasks, including Multi-genre Natural Language Inference (MNLI, Williams et al. (2017)) and Question Natural Language Inference (QNLI, Wang et al. (2018)), as well as a language modelling task on a large corpus and a small one, (WikiText-103 and WikiText-2 (Merity et al., 2018)), are…”
Representation Degeneration Problem in Training Natural Language Generation Models (2019, background): “Furthermore, it is interesting to combine with other approaches, e.g. (Gong et al., 2018), to enrich the representation of word embeddings.”
MaxUp: A Simple Way to Improve Generalization of Neural Network Training (2020, unknown): “50 AWD-LSTM + FRAGE (Gong et al., 2018) 24M 58.”
Calibration, Entropy Rates, and Memory in Language Models (2019, background): “Recent advances in language modeling have resulted in significant improvements on a wide variety of benchmarks (Dai et al., 2018; Gong et al., 2018; Takase et al., 2018).”
Learning Distributed Representations of Symbolic Structure Using Binding and Unbinding Operations (2018, unknown): “on the dev test, so we omitted the results from GRU in the table. model WikiText-103 WikiText-2 #Params Two-layer Architecture - dim 1024 TPRU 41.4 76.8 12.6m LSTM 42.3 76.0 16.8m SOTA 16.4 [17] 39.1 [10] 257m / 35m (a) 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 Epochs 3.5 4.0 4.5 5.0 5.5 6.0 Negative Log-”
Sentence-wise Smooth Regularization for Sequence to Sequence Learning (2018, background): “ility tokens. Besides, we wonder where the non-smooth problem comes from and then study the components of tokens with low probability in the train set. As is well-known, rare words are hard to learn (Gong et al. 2018) and low probabilities may just come from rare words. However, as shown in Figure 1”
Linguistically-Informed Specificity and Semantic Plausibility for Dialogue Generation (2019, background): “Learning meaningful representations for infrequent words is a known challenge (Gong et al., 2018) hence low-quality representations may increase the probability of the sentence being implausible.”