The Evolved Transformer

By David R. So, Chen Liang, Quoc V. Le. 2019

499 citations recorded by the source corpus.

Recent works have highlighted the strength of the Transformer architecture on sequence tasks while, at the same time, neural architecture search (NAS) has begun to outperform human-designed models. Our goal is to apply NAS to search for a better alternative to the Transformer. We first construct a large search space inspired by the recent advances in feed-forward sequence models and then run evolutionary architecture search with warm starting by seeding our initial population with the Transformer. To directly search on the computationally expensive WMT 2014 English-German translation task, we develop the Progressive Dynamic Hurdles method, which allows us to dynamically allocate more resources to more promising candidate models. The architecture found in our experiments -- the Evolved Transformer -- demonstrates consistent improvement over the Transformer on four well-established language tasks: WMT 2014 English-German, WMT 2014 English-French, WMT 2014 English-Czech and LM1B. At a big model size, the Evolved Transformer establishes a new state-of-the-art BLEU score of 29.8 on WMT'14 English-German; at smaller sizes, it achieves the same quality as the original "big" Transformer with 37.6% less parameters and outperforms the Transformer by 0.7 BLEU at a mobile-friendly model size of 7M parameters.

Subfields: Electronic, Optical and Magnetic Materials, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Electronic, Optical and Magnetic Materials

In Model Architectures

Citations in the corpus

Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, background): “…(Tay et al., 2020c; Kitaev et al., 2020; Choromanski et al., 2020; Tay et al., 2020b; Ainslie et al., 2020; Jaegle et al., 2021), architecture search (So et al., 2019), alternative non Transformer-based architectures (Tolstikhin et al., 2021; Tay et al., 2021a; 2020a; Lee-Thorp et al., 2021).”
Carbon Emissions and Large Neural Network Training (2021, methodology): “Figure 2 plots the number of parameters on the X axis and number of total FLOPS on the Y axis relative to Transformer (Big) [So19] using a log-log graph.”
Towards a Human-like Open-Domain Chatbot (2020, methodology): “We vary the number of layers, attention heads, total training steps, whether we use Evolved Transformer or regular Transformer and whether we train with hard labels or soft labels/distillation (Hinton et al., 2015).”
Attention Augmented Convolutional Networks (2019, methodology): “lf-attention in cooperation with convolutions is a theme shared by recent work in Natural Language Processing [49] and Reinforcement Learning [52]. For example, the QANet [50] and Evolved Transformer [38] architectures alternate between self-attention layers and convolution layers for Question Answe”
Energy and Policy Considerations for Modern Deep Learning Research (2020, methodology): “So, Liang, and Le (2019) report that NAS achieves a new state-of-the-art BLEU score of 29.7 for English to German machine translation, an increase of just 0.1 BLEU at the cost of at least $150k in on-demand compute time and potentially substantial carbon emissions.”
Primer: Searching for Efficient Transformers for Language Modeling (2021, methodology): “2 for how we employ relative dimensions [13] to resize our models.”
Energy and Policy Considerations for Deep Learning in NLP (2019, methodology): “So et al. (2019) report that NAS achieves a new state-of-the-art BLEU score of 29.7 for English to German machine translation, an increase of just 0.1 BLEU at the cost of at least $150k in on-demand compute time and non-trivial carbon emissions.”
Do Transformer Modifications Transfer Across Implementations and Applications? (2021, methodology): “The Evolved Transformer (So et al., 2019) was designed via evolution-based architecture search (Real et al., 2019) where the initial population was seeded with the original Transformer.”
Efficient Transformers: A Survey (2020, background): “As such, it is only natural that a wealth of research has been dedicated to making fundamental improvements to the model over the past few years (Dehghani et al., 2018; So et al., 2019; Ahmed et al., 2017).”
Weight Agnostic Neural Networks (2019, methodology): “In order to achieve state-of-the-art results, recent methods narrow the search space to architectures composed of basic building blocks with strong domain priors such as CNNs [64, 72, 89, 119], recurrent cells [48, 72, 119] and self-attention [100].”
Are Labels Necessary for Neural Architecture Search? (2020, background): “There are special focuses on reducing the overall time cost of the search process [19,24,20], or on extending to a larger variety of tasks [4,18,11,28].”
AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, methodology): “Second, constrained search spaces need to be carefully composed (Zoph et al., 2018; So et al., 2019; Negrinho et al., 2019), thus creating a new burden on researchers and undermining the purported objective of saving their time.”
Semi-Supervised Neural Architecture Search (2020, methodology): “Neural architecture search (NAS) for automatic architecture design has been successfully applied in several tasks including image classification and language modeling [42, 26, 6].”
Continual and Multi-Task Architecture Search (2019, background): “Elsken et al. (2019) and So et al. (2019) proposed evolutionary architecture search algorithms that dynamically allocate more resources for promising architecture candidates, but these works are different from us in that they do not consider the case where we have continual incoming-data from…”
A Survey on Neural Architecture Search (2019, background): “, 2018), transformers (So et al., 2019) and graph CNNs (Gao et al.”
NAS-Bench-101: Towards Reproducible Neural Architecture Search (2019, neutral): “This has resulted in state of the art performance in the do-* main of image classiﬁcation (Zoph et al., 2018; Real et al., 2018; Huang et al., 2018), and has shown promising results in other domains, such as sequence modeling (Zoph & Le, 2016; So et al., 2019).”
Evaluating the Search Phase of Neural Architecture Search (2019, background): “In other words, the search and robustness of these algorithms are generally not studied, with (Liu et al., 2019b; So et al., 2019) the only exception for robustness, where results obtained with different random seeds were reported.”
Evolutionary-Neural Hybrid Agents for Architecture Search (2018, background): “Recent work has shown that evolutionary agents for architecture search can match or outperform deep RL methods [33, 37].”
Transformers in Vision: A Survey (2021, background): “While Nerual Architecuter Search (NAS) has been well explored for CNNs to ind an optimized architecture, it is relatively less explored in Transformers (even for language transformers [179, 205]).”