NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing

By Nikita Klyuchnikov, Ilya Trofimov, Ekaterina Artemova, Mikhail Salnikov, Maxim V. Fedorov, Evgeny Burnaev. 2020

135 citations recorded by the source corpus.

Neural Architecture Search (NAS) is a promising and rapidly evolving research area. Training a large number of neural networks requires an exceptional amount of computational power, which makes NAS unreachable for those researchers who have limited or no access to high-performance clusters and supercomputers. A few benchmarks with precomputed neural architectures performances have been recently introduced to overcome this problem and ensure reproducible experiments. However, these benchmarks are only for the computer vision domain and, thus, are built from the image datasets and convolution-derived architectures. In this work, we step outside the computer vision domain by leveraging the language modeling task, which is the core of natural language processing (NLP). Our main contribution is as follows: we have provided search space of recurrent neural networks on the text datasets and trained 14k architectures within it; we have conducted both intrinsic and extrinsic evaluation of the trained models using datasets for semantic relatedness and language understanding evaluation; finally, we have tested several NAS algorithms to demonstrate how the precomputed results can be utilized. We consider that the benchmark will provide more reliable empirical findings in the community and stimulate progress in developing new NAS methods well suited for recurrent architectures.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Arber Zela
2. Julien N. Siems
3. Margret Keuper
4. Jovita Lukasik
5. Lucas Zimmer
6. Frank Hutter
7. Chuang Niu
8. Chen Wei

In Computer Vision and Pattern Recognition

1. Arber Zela
2. Julien N. Siems
3. Margret Keuper
4. Jovita Lukasik
5. Lucas Zimmer
6. Frank Hutter
7. Nicholas Donald Lane
8. Chen Wei

In Natural Language Processing

1. Frank Hutter
2. Nicholas Donald Lane
3. Margret Keuper

In Neural Architecture Search & AutoML

1. Arber Zela
2. Julien N. Siems
3. Jovita Lukasik
4. Margret Keuper
5. Lucas Zimmer
6. Frank Hutter
7. Nicholas Donald Lane
8. Chen Wei

Citations in the corpus

• NPENAS: Neural Predictor Guided Evolution for Neural Architecture Search (2022, methodology): “…extensive experiments on five search spaces, including NASBench-101 [1], NASBench-201 [9], and DARTS [10] for image classification, NASBench-NLP [11] for natural language processing (NLP), and NASBench-ASR [12] for speech recognition tasks, to demonstrate the superiority of NPENAS over existing…”
• Zero-Cost Proxies for Lightweight NAS (2021, background): “• NAS-Bench-NLP: Klyuchnikov et al. (2020) investigate the architectures of 14k different recurrent cells in natural language processing (NLP) tasks such as next word prediction.”
• NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search (2020, methodology): “NAS-Bench-NLP (Klyuchnikov et al., 2020) was recently proposed as a tabular benchmark for NAS in the Natural Language Processing domain.”