FastText.zip: Compressing text classification models

By Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Herv'e J'egou, Tomas Mikolov. 2016

1,352 citations recorded by the source corpus.

We consider the problem of producing compact architectures for text classification, such that the full model fits in a limited amount of memory. After considering different solutions inspired by the hashing literature, we propose a method built upon product quantization to store the word embeddings. While the original technique leads to a loss in accuracy, we adapt this method to circumvent the quantization artifacts. As a result, our approach produces a text classifier, derived from the fastText approach, which at test time requires only a fraction of the memory compared to the original one, without noticeably sacrificing the quality in terms of classification accuracy. Our experiments carried out on several benchmarks show that our approach typically requires two orders of magnitude less memory than fastText while being only slightly inferior with respect to accuracy. As a result, it outperforms the state of the art by a good margin in terms of the compromise between memory usage and accuracy.

Subfields: Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

Toolformer: Language Models Can Teach Themselves to Use Tools (2023, methodology): “The source language is automatically detected using the fastText classifier (Joulin et al., 2016), while the target language is always set to English.”
Neural Machine Translation for English–Kazakh with Morphological Segmentation and Synthetic Data (2019, methodology): “Following Artetxe and Schwenk (2018a), we removed sentences shorter than 3 words and longer than 80 words, and sentence pairs where either sentence is classiﬁed as another language us-ing the FastText language identiﬁer (Joulin et al., 2016a,b).”
CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data (2019, methodology): “Following Grave et al. (2018), we use the language classiﬁer from fastText (Joulin et al., 2016b; Grave et al., 2018).”
Unbounded cache model for online language modeling with open vocabulary (2017, methodology): “In the context of text, several works have shown that compression does not signiﬁcantly reduce the performance of models [17, 24, 30].”
HTLM: Hyper-Text Pre-Training and Prompting of Language Models (2021, methodology): “We also ﬁlter out data to only English ( en ) after 170k steps us-ing FastText (Joulin et al., 2016).”
Parallel Corpus Filtering via Pre-trained Language Models (2020, methodology): “We adopt the fastText (Joulin et al., 2017, 2016) language identiﬁcation toolkit in our language detection ﬁlter.”
Reducing Transformer Depth on Demand with Structured Dropout (2019, methodology): “, 2016), text classification (Joulin et al., 2016) and language modeling (Murray & Chiang, 2015).”
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “This dataset is extracted from the Common Crawl (CC) using a fastText-based classifier (Joulin et al., 2016).”
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, methodology): “It is deduplicated and follows the same approach as Grave et al. (2018) by using FastText LangID (Joulin et al., 2016, 2017) on a line-level.”
An Analysis of Encoder Representations in Transformer-Based Machine Translation (2018, methodology): “5https://paracrawl.eu/ 6We used the fasttext language identifier tool (Joulin et al., 2016b,a) from https://fasttext.cc/docs/ en/language-identification.html 7We used the SACREBLEU script (Post, 2018), with signature BLEU+case.mixed+lang.en{targetLanguage}+numrefs.1+smooth.exp+test.wmt{17,18}+…”
Pre-training via Paraphrasing (2020, methodology): “Data Pre-processing We de-duplicate the data, and identify languages using FastText [Joulin et al., 2016].”
Training with Quantization Noise for Extreme Model Compression (2020, methodology): “Other quantization methods such as Vector Quantization (VQ) and PQ [36] quantize blocks of weights simulatneously to achieve higher compression rate [8, 37, 38, 39].”
ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing (2020, neutral): “Newer contextual models improved both over previous LM-based approaches [17] (3-5 percentage points in Q3) and over non-contextualized word2vec-type approaches [64], [65], [66] (12-17 percentage points in Q3).”
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages (2020, methodology): “…et al. (2019) reproduced the pipeline proposed by ( Grave et al., 2018) language classiﬁcation was performed using the fastText linear classiﬁer (Joulin et al., 2016, 2017), which was trained by Grave et al. (2018) to recognize 176 languages and was shown to have an extremely good accuracy to…”
CamemBERT: a Tasty French Language Model (2019, methodology): “OS-CAR follows the same approach as (Grave et al., 2018) by using a language classiﬁcation model based on the fastText linear classiﬁer (Joulin et al., 2016; Grave et al., 2017) pre-trained on Wikipedia, Tatoeba and SETimes, which supports 176 different languages.”
Block Pruning For Faster Transformers (2021, background): “In contrast, structured pruning removes coherent groups of weights (Murray and Chiang, 2015; See et al., 2016; Joulin et al., 2016; Fan et al., 2020; Sajjad et al., 2020).”
ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning (2021, background): “Ultimately, joining the strengths of three different, yet complementary, ﬁelds (HPC, NLP and computational biology) affected the advance.”
Few-shot Text Classification with Distributional Signatures (2019, methodology): “3 IMPLEMENTATION DETAILS We use pre-trained fastText embeddings (Joulin et al., 2016) for our model and all baselines.”
Citation Detective : a Public Dataset to Improve and Quantify Wikipedia Citation Quality at Scale (2020, methodology): “An input instance for the Citation Need model is a made of (1) set of FasTtext [4] word vectors representing the each word in a sentence, and (2) the average word vector for all the words in the section title where the sentence lies (see [7] for more details).”