Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP

By Haonan Yu, Sergey Edunov, Yuandong Tian, Ari S. Morcos. 2019

157 citations recorded by the source corpus.

The lottery ticket hypothesis proposes that over-parameterization of deep neural networks (DNNs) aids training by increasing the probability of a "lucky" sub-network initialization being present rather than by helping the optimization process (Frankle & Carbin, 2019). Intriguingly, this phenomenon suggests that initialization strategies for DNNs can be improved substantially, but the lottery ticket hypothesis has only previously been tested in the context of supervised learning for natural image tasks. Here, we evaluate whether "winning ticket" initializations exist in two different domains: natural language processing (NLP) and reinforcement learning (RL).For NLP, we examined both recurrent LSTM models and large-scale Transformer models (Vaswani et al., 2017). For RL, we analyzed a number of discrete-action space tasks, including both classic control and pixel control. Consistent with workin supervised image classification, we confirm that winning ticket initializations generally outperform parameter-matched random initializations, even at extreme pruning rates for both NLP and RL. Notably, we are able to find winning ticket initializations for Transformers which enable models one-third the size to achieve nearly equivalent performance. Together, these results suggest that the lottery ticket hypothesis is not restricted to supervised learning of natural images, but rather represents a broader phenomenon in DNNs.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “ed to other tasks, including scientiﬁc document retrieval [Boudin et al., 2020], creating artiﬁcial in-domain retrieval data [Ma et al., 2020], and helping users in ﬁnding answers in product reviews [Yu et al., 2020b]. Document expansion with doc2query, in effect, shifts computationally expensive in”
Linear Mode Connectivity and the Lottery Ticket Hypothesis (2019, background): “Other efforts use rewinding to further study lottery tickets (Yu et al., 2020; Frankle et al., 2020; Caron et al., 2020; Savarese et al., 2020; Yin et al., 2020).”
Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning (2020, methodology): “14 We might consider ﬁnding a lottery ticket for BERT, which we would expect to ﬁt the GLUE training data just as well as pre-trained BERT (Morcos et al., 2019; Yu et al., 2019).”
When BERT Plays the Lottery, All Tickets Are Winning (2020, background): “, 2019), and exist both in LSTM and Transformer models for NLP (Yu et al., 2020).”
Movement Pruning: Adaptive Sparsity by Fine-Tuning (2020, methodology): “Recent work also builds upon iterative magnitude pruning with rewinding [Yu et al., 2020] to train sparse language models from scratch.”
The Lottery Ticket Hypothesis for Pre-trained BERT Networks (2020, background): “Although the lottery ticket hypothesis has been evaluated in the context of NLP [17, 18] and trans-formers [17, 23], it remains poorly understood in the context of pre-trained BERT models.”
FastFormers: Highly Efficient Transformer Models for Natural Language Understanding (2020, unknown): “Furthermore, there are various papers published to apply this pruning strategy to Transformer models including: (Yu et al., 2019; Sanh et al., 2020; Gordon et al., 2020).”
MLPruning: A Multilevel Structured Pruning Framework for Transformer-based Models (2021, background): “Most of the previous pruning work in NLP has focused on unstructured pruning [36, 7, 50, 31, 57], rather than structured pruning [27, 11].”
TX-Ray: Quantifying and Explaining Model-Knowledge Transfer in (Un-)Supervised NLP (2019, background): “We also find, that TX-Ray guided pruning of supervision task (ir-)relevant neuron-knowledge (RQ3) can identify ‘lottery ticket’-like [9, 40] model components that concentrate model performance or improve robustness.”
Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecture (2020, unknown): “For high sparsity values, SLT-MP models are also superior to the SLT models by Yu et al. (2020), even though they start of from a better performing baseline.”
Playing Lottery Tickets with Vision and Language (2021, background): “The properties of LTH are widely studied for image classification [44, 15, 18, 58, 68, 74, 47], and recently, the observations of LTH have also been evidenced across other fields, such as natural language processing [19, 76, 54, 6, 9], object detection [21], generative adversarial networks [10, 31, ”
Evaluating Lottery Tickets Under Distributional Shifts (2019, background): “Evidence of the existence of winning tickets has been empirically shown on a range of tasks, including computer vision, reinforcement learning, and natural language processing (Frankle and Carbin, 2019; Yu et al., 2019).”
Learned Token Pruning for Transformers (2021, methodology): “In NLP, for unstructured pruning, the lottery-ticket hypothesis [16] has been explored for transformers in [7, 34, 55].”