Evaluating Lottery Tickets Under Distributional Shifts

By Shrey Desai, Hongyuan Zhan, Ahmed Aly. 2019

42 citations recorded by the source corpus.

The Lottery Ticket Hypothesis suggests large, over-parameterized neural networks consist of small, sparse subnetworks that can be trained in isolation to reach a similar (or better) test accuracy. However, the initialization and generalizability of the obtained sparse subnetworks have been recently called into question. Our work focuses on evaluating the initialization of sparse subnetworks under distributional shifts. Specifically, we investigate the extent to which a sparse subnetwork obtained in a source domain can be re-trained in isolation in a dissimilar, target domain. In addition, we examine the effects of different initialization strategies at transfer-time. Our experiments show that sparse subnetworks obtained through lottery ticket training do not simply overfit to particular domains, but rather reflect an inductive bias of deep neural networks that can be exploited in multiple domains.

Subfields: Artificial Intelligence, Efficiency & Compression, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

In Safety, Robustness & Interpretability

Citations in the corpus

Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, background): “Another recent line of work has shown that pretrained models such as BERT are redundant in their capacity, allowing for significant sparsification without much degradation in end metrics [45, 46, 47].”
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (2020, background): “For example, a recent line of work has shown that pre-trained models such as BERT are redundant in their capacity, allowing for significant sparsification without much degradation in end metrics (Chen et al., 2020; Prasanna et al., 2020; Desai et al., 2019).”
Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning (2021, background): “(Mehta, 2019; Morcos et al., 2019; Desai et al., 2019) pioneer to study the transferability of the ticket identified on one source task to another target task, which delivers insights on one-shot transferability of LTH.”
Calibration of Pre-trained Transformers (2020, unknown): “In contrast, BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) are large-scale, pre-trained language models with stacked, general purpose Transformer (Vaswani et al., 2017) layers. ness using sentiment analysis (Chen et al., 2018; Peng et al., 2018; Miller, 2019; Desai et al., 2019).”