Enriching Word Vectors with Subword Information
By Piotr Bojanowski, Edouard Grave, Armand Joulin, Tomas Mikolov. 2016
10,838 citations recorded by the source corpus.
Continuous word representations, trained on large unlabeled corpora are useful for many natural language processing tasks. Popular models that learn such representations ignore the morphology of words, by assigning a distinct vector to each word. This is a limitation, especially for languages with large vocabularies and many rare words. In this paper, we propose a new approach based on the skipgram model, where each word is represented as a bag of character n-grams. A vector representation is associated to each character n-gram; words being represented as the sum of these representations. Our method is fast, allowing to train models on large corpora quickly and allows us to compute word representations for words that did not appear in the training data. We evaluate our word representations on nine different languages, both on word similarity and analogy tasks. By comparing to recently proposed morphological word representations, we show that our vectors achieve state-of-the-art performance on these tasks.
Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Graham Neubig
- Jianfeng Gao
- Noah A. Smith
- Yejin Choi
- Kyunghyun Cho
In Natural Language Processing
- Mohit Bansal
- Graham Neubig
- Luke S. Zettlemoyer
- Yejin Choi
- Percy Liang
- Jianfeng Gao
- Noah A. Smith
- Kyunghyun Cho
In Representation & Self-Supervised Learning
- Sebastian Ruder
- Graham Neubig
- Ivan Vulic
- Hinrich Schütze
- Yoshua Bengio
- Iryna Gurevych
- Samuel R. Bowman
- Percy Liang
Citations in the corpus
- Deep Contextualized Word Representations (2018, methodology): “Previously proposed methods overcome some of the shortcomings of traditional word vectors by either enriching them with subword information (e.g., Wieting et al., 2016; Bojanowski et al., 2017) or learning separate vectors for each word sense (e.”
- Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, background): “Distributed representations of words (or word embeddings) (Bengio et al., 2003; Collobert et al., 2011; Mikolov et al., 2013; Pennington et al., 2014; Bojanowski et al., 2016) have shown to provide useful features for various tasks in natural language processing and computer vision.”
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “Our analysis uses the fastText (Bojanowski et al., 2016) language identification tool and a threshold of 0.”
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “For training, we apply byte-pair encoding tokenization (Bojanowski et al., 2017; Radford et al., 2019b) and randomly split this dataset into a 29:1 ratio to obtain training (168.2 GB) and validation (5.8 GB) sets, respectively.”
- Cross-lingual Language Model Pretraining (2019, background): “[25] showed that using a shared vocabulary between two languages and then applying fastText [6] on the concatenation of their monolingual corpora also directly provides high-quality cross-lingual word embeddings (Concat) for languages that share a common alphabet.”
- Meta-Learning for Low-Resource Neural Machine Translation (2018, methodology): “We then estimate word vectors using fast-Text (Bojanowski et al., 2016) and align them across all the languages in an unsupervised way using MUSE (Conneau et al., 2018) to get multilingual word vectors.”
- CoQA: A Conversational Question Answering Challenge (2018, methodology): “, 2014) for conversational models and fastText (Bojanowski et al., 2017) for reading comprehension models, based on empirical performance.”
- Learning Word Vectors for 157 Languages (2018, methodology): “Using this data, we trained word vectors using an extension of the fastText model with subword information (Bojanowski et al., 2017), as described in Section 3.”
- Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, methodology): “Somewhat surprisingly, in Table 3 we observe that the learned word embeddings are competitive with popular methods such as GloVe, word2vec, and fasttext (Bojanowski et al., 2016) on the benchmarks presented by Faruqui & Dyer (2014) and Tsvetkov et al. (2015).”
- RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, methodology): “We first filter non-English text with FASTTEXT (Bojanowski et al., 2016).”
- ELI5: Long Form Question Answering (2019, methodology): “The data is tokenized with Spacy 5 and we select English documents with FastText language identification (Bojanowski et al., 2017).”
- Hierarchical Neural Story Generation (2018, methodology): “each prompt was created using FASTTEXT (Bojanowski et al., 2016) and FAISS (Johnson et al.”
- TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, positive): “An offline OCR detector [10] detects scene text regions and represents the region with its visual, coordinates, FastText [9], and Pyramidal Histogram of Characters (PHOC) [2] features.”
- Learning from Dialogue after Deployment: Feed Yourself, Chatbot! (2019, background): “Tokens are embedded with fastText (Bojanowski et al., 2017) 300-dimensional embeddings.”
- Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, methodology): “Our word embeddings are initialized with FastText (Bojanowski et al., 2016) to capture the sub-word features and then fine-tuned.”
- Generating Datasets with Pretrained Language Models (2021, neutral): “There are many unsupervised approaches to obtaining sentence embeddings, for example by averaging word embeddings (Mikolov et al., 2013; Pennington et al., 2014; Bojanowski et al., 2017) or with carefully designed sentence-level objectives (Le and Mikolov, 2014; Kiros et al., 2015).”
- Unsupervised Question Decomposition for Question Answering (2020, methodology): “sub-questions, we embed any text t into a vector vt by summing the FastText vectors (Bojanowski et al., 2017)3 for words in t and use cosine as our similarity metric f .”
- ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition (2021, methodology): “A weighted average of all nouns (NN, NNS) in the label was computed (each represented as a FastText [3] embedding)5 where each weight was the (normalized) frequency of the noun across all nouns in the dataset.”