Natural Language Processing with Small Feed-Forward Networks
By Jan A. Botha, Emily Pitler, Ji Ma, Anton Bakalov, Alexandru Salcianu, David Weiss, Ryan T. McDonald, Slav Petrov. 2017
39 citations recorded by the source corpus.
We show that small and shallow feed-forward neural networks can achieve near state-of-the-art results on a range of unstructured and structured language processing tasks while being considerably cheaper in memory and computational requirements than deep recurrent models. Motivated by resource-constrained environments like mobile phones, we showcase simple techniques for obtaining such small neural network models, and investigate different tradeoffs when deciding how to allocate a small memory budget.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Graham Neubig
- Orhan Firat
- Sebastian Gehrmann
- Ankur Bapna
- Luke S. Zettlemoyer
- Julia Kreutzer
- Iryna Gurevych
- Aditya Siddhant
Citations in the corpus
- Federated Learning for Emoji Prediction in a Mobile Keyboard (2019, background): “The logs are filtered further to only include sentences that are labeled as English with high confidence by a language detection model (Botha et al., 2017; Zhang et al., 2018).”
- Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, methodology): “On this task, the CLD3 classifier6 achieves an average precision of only 40.6%, illustrating the issues with LangID on web domain data (Caswell et al., 2020).”
- Dialect-robust Evaluation of Generated Text (2022, methodology): “We split the corpus into sentences and use a Language Identification (LangID) model (Zhang et al., 2018) by Botha et al. (2017) to identify the language and locale information for the sentences.7 Besides LangID output, mC4 provides the URL where a sentence originated from which we extract the…”
- A Fast, Compact, Accurate Model for Language Identification of Codemixed Text (2018, methodology): “Simple feed-forward networks have achieved near state-of-the-art performance in a wide range of NLP tasks (Botha et al., 2017; Weiss et al., 2015).”
- High Performance Natural Language Processing (2020, unknown): “Case studies: Botha et al. (2017); So et al. (2019); Sun et al. (2020); Yan et al. (2020); Wang et al. (2020a); Iandola et al. (2020); Mehta et al. (2020); Reimers and Gurevych (2019); Khandelwal et al.”