Bag of Tricks for Efficient Text Classification
By Armand Joulin, Edouard Grave, Piotr Bojanowski, Tomas Mikolov. 2016
5,082 citations recorded by the source corpus.
This paper explores a simple and efficient baseline for text classification. Our experiments show that our fast text classifier fastText is often on par with deep learning classifiers in terms of accuracy, and many orders of magnitude faster for training and evaluation. We can train fastText on more than one billion words in less than ten minutes using a standard multicore CPU, and classify half a million sentences among 312K classes in less than a minute.
Subfields: Artificial Intelligence, Natural Language Processing, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Yejin Choi
- Graham Neubig
- Noah A. Smith
- Yoshua Bengio
In Natural Language Processing
- Mohit Bansal
- Luke S. Zettlemoyer
- Graham Neubig
- Yejin Choi
- Jason Aaron Edward Weston
- Jianfeng Gao
- Noah A. Smith
- Kyunghyun Cho
In Efficiency & Compression
- Alexander M. Rush
- Angela Fan
- Armand Joulin
- Edouard Grave
- Herv'e J'egou
- Luke S. Zettlemoyer
- Benoît Sagot
- Pedro Ortiz Suarez
Citations in the corpus
- SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, background): “fastText (Joulin et al., 2017): This library models a single span of text as a bag of n-grams, and tries to predict the probability of an ending being correct or incorrect independently.”
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, methodology): “(2018), training fastText classifiers (Joulin et al., 2016) to predict entailment labels on SNLI and MultiNLI using only the hypothesis as input.”
- SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, neutral): “8 FastText Skip-gram with sub-word character n-grams (Joulin et al., 2016) 65.”
- Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, methodology): “All three use FastText (Joulin et al., 2016) pre-trained word embeddings2 to which we apply a Highway transformation (Srivastava et al.”
- Defending Against Neural Fake News (2019, methodology): “We perform domain adaptation to adapt BERT to the news domain, as well as to account for long articles; details in Appendix C. d. FastText, an o↵-the-shelf library for bag-of-ngram text classification (Joulin et al., 2017).”
- HellaSwag: Can a Machine Really Finish Your Sentence? (2019, methodology): “We consider three variations: GloVe embeddings, ELMo embeddings, or (frozen) BERT-Base embeddings.9 e. FastText: (Joulin et al., 2017) An off-the-shelf library for bag-of-words text classification.10 We compare all models to human performance by asking five independent crowd workers to solve the…”
- Annotation Artifacts in Natural Language Inference Data (2018, methodology): “Specifically, we use fastText (Joulin et al., 2017), an off-the-shelf text classifier that models text as a bag of words and bigrams, to predict the entailment label of the hypothesis.”
- Learning Word Vectors for 157 Languages (2018, methodology): “For this purpose, we built a fast language detector using the fastText linear classifier (Joulin et al., 2017), which can recognize 176 languages.”
- Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function (2019, background): “Later, better word representations were introduced, such as latent semantic analysis (Deer-wester et al. 1990), skipgram (Mikolov et al. 2013), and fastText (Joulin et al. 2017), which improved classification accuracy.”
- Neural Machine Translation for English–Kazakh with Morphological Segmentation and Synthetic Data (2019, methodology): “Following Artetxe and Schwenk (2018a), we removed sentences shorter than 3 words and longer than 80 words, and sentence pairs where either sentence is classified as another language us-ing the FastText language identifier (Joulin et al., 2016a,b).”
- CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data (2019, methodology): “Following Grave et al. (2018), we use the language classifier from fastText (Joulin et al., 2016b; Grave et al., 2018).”
- Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, methodology): “…commonly employed neural networks like CNNs, RNNs, and LSTMs (Georgakopoulos et al., 2018; van Aken et al., 2018; Gunasekara and Nejadgholi, 2018; Joulin et al., 2017; Kshirsagar et al., 2018; Mishra et al., 2018; Mitrovi´c et al., 2019; Sigurbergsson and Derczynski, 2020), contemporary…”
- Galactica: A Large Language Model for Science (2022, methodology): “For Scientific Common Crawl, we train a fasttext classifier to identify Common Crawl webpages with scientific content (Joulin et al., 2016) using a noisy set of 600 domains.”
- “I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, positive): “We then expanded these terms by selecting additional relevant terms from among the 50 nearest neighbors per existing term as measured with fastText word embeddings (Joulin et al., 2017), as well as WordNet synonyms and antonyms (Fellbaum and Miller, 1998).”
- Finding Generalizable Evidence by Learning to Convince Q&A Models (2019, methodology): “…define two model classes that select the answer most similar to the input passage sentences: L(i) = TFIDF(S, [Q;A(i)]), and L(i) = TFIDF(S,A(i)). fastText We define a function BoWFT that computes the average bag-of-words representation of some text using fastText embeddings (Joulin et al., 2017).”
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, methodology): “Other datasets using internet data include Food101 [80], where the goal is to predict the dish of recipes and images; various versions of Yelp reviews [52]; Walmart and Ferramenta product classification [90, 21]; social media name tagging (Twitter and Snapchat) [49]; social media target-oriented sen”
- Towards VQA Models That Can Read (2019, methodology): “OCR tokens are first embedded using pretrained FastText embeddings (fO) [19], which can generate word embeddings even for OOV tokens as explained in [19].”
- Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack (2019, neutral): “We also compare it the fastText classifier (Joulin et al., 2017) for which a given sentence is encoded as the average of individual word vectors that are pre-trained on a large corpus issued from Wikipedia.”