Drawing early-bird tickets: Towards more efficient training of deep networks

By Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Yingyan Lin, Zhangyang Wang, Richard Baraniuk. 2019

285 citations recorded by the source corpus.

(Frankle & Carbin, 2019) shows that there exist winning tickets (small but critical subnetworks) for dense, randomly initialized networks, that can be trained alone to achieve comparable accuracies to the latter in a similar number of iterations. However, the identification of these winning tickets still requires the costly train-prune-retrain process, limiting their practical benefits. In this paper, we discover for the first time that the winning tickets can be identified at the very early training stage, which we term as early-bird (EB) tickets, via low-cost training schemes (e.g., early stopping and low-precision training) at large learning rates. Our finding of EB tickets is consistent with recently reported observations that the key connectivity patterns of neural networks emerge early. Furthermore, we propose a mask distance metric that can be used to identify EB tickets with low computational overhead, without needing to know the true winning tickets that emerge after the full training. Finally, we leverage the existence of EB tickets and the proposed mask distance to develop efficient training methods, which are achieved by first identifying EB tickets via low-cost schemes, and then continuing to train merely the EB tickets towards the target accuracy. Experiments based on various deep networks and datasets validate: 1) the existence of EB tickets, and the effectiveness of mask distance in efficiently identifying them; and 2) that the proposed efficient training via EB tickets can achieve up to 4.7x energy savings while maintaining comparable or even better accuracy, demonstrating a promising and easily adopted method for tackling cost-prohibitive deep network training.

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

1. Tri Dao
2. Yen-Chun Chen
3. Christopher R'e
4. Atri Rudra
5. Jingjing Liu
6. Yu Cheng
7. Zhe Gan
8. Beidi Chen

Citations in the corpus

• Rethinking Attention with Performers (2020, background): “, 2019) and lower energy consumption (You et al., 2020), as regular Transformers require very large computational resources.”
• Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers (2020, background): “…As we have shown, Performers with FAVOR are characterized by much lower compute costs and substantially lower space complexity which can be directly translated to CO 2 emission reduction [42] and lower energy consumption [50], as regular Transformers require very large computational resources.”
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training (2021, background): “, 2018a) or a partial converged training time (partial dense over-parameterization) (Louizos et al., 2017; Zhu & Gupta, 2017; Gale et al., 2019; Savarese et al., 2019; Kusupati et al., 2020; You et al., 2019).”
• Structured Pruning for Deep Convolutional Neural Networks: A Survey (2023, background): “Early Bird (EB) [153] proposes to find the winning ticket in the early stages of training, rather than fully training the dense network.”
• On the Effectiveness of Parameter-Efficient Fine-Tuning (2022, background): “…works (Salman et al., 2020; Jiang et al., 2020; Li and Zhang, 2021; Hua et al., 2021) propose to train model adversarially while some researchers (You et al., 2019; Liang et al., 2021; Ansell et al., 2021; Chen et al., 2021) propose to utilize the lottery ticket approaches to prune the network.”
• Monarch: Expressive Structured Matrices for Efficient and Accurate Training (2022, result): “…comparable results with the specialized lottery tickets; Frankle et al. (2019) improves the stability of the lottery tickets by iterative pruning; You et al. (2019); Chen et al. (2021) attempt to make it efficient to find lottery tickets; Frankle et al. (2020) found that subnetworks reach full…”
• Pruning neural networks without any data by iteratively conserving synaptic flow (2020, methodology): “Magnitude pruning effectively prunes parameters based on the variance of their initialization, which for common network initializations, such as Xavier [36] or Kaiming [37], are inversely proportional to the width of a layer [33].”
• Playing Lottery Tickets with Vision and Language (2021, background): “The properties of LTH are widely studied for image classification [44, 15, 18, 58, 68, 74, 47], and recently, the observations of LTH have also been evidenced across other fields, such as natural language processing [19, 76, 54, 6, 9], object detection [21], generative adversarial networks [10, 31, ”
• Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning (2021, background): “…(Frankle & Carbin, 2019; Liu et al., 2019; Wang et al., 2020; Evci et al., 2019; Frankle et al., 2020; Savarese et al., 2020; Yin et al., 2020; You et al., 2020), LTH has been explored widely in numerous contexts, such as natural language processing (Gale et al., 2019), reinforcement learning…”