Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing

By Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, Lawrence Carin. North American Chapter of the Association for Computational Linguistics, 2019

453 citations recorded by the source corpus.

Variational autoencoders (VAE) with an auto-regressive decoder have been applied for many natural language processing (NLP) tasks. VAE objective consists of two terms, the KL regularization term and the reconstruction term, balanced by a weighting hyper-parameter \beta. One notorious training difficulty is that the KL term tends to vanish. In this paper we study different scheduling schemes for \beta, and show that KL vanishing is caused by the lack of good latent codes in training decoder at the beginning of optimization. To remedy the issue, we propose a cyclical annealing schedule, which simply repeats the process of increasing \beta multiple times. This new procedure allows us to learn more meaningful latent codes progressively by leveraging the results of previous learning cycles as warm re-restart. The effectiveness of cyclical annealing schedule is validated on a broad range of NLP tasks, including language modeling, dialog response generation and semi-supervised text classification.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/845cafab91da3441d56a8740f0c8038916863dcb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Prompt Conditioned VAE: Enhancing Generative Replay for Lifelong Learning in Task-Oriented Dialogue (2022, methodology): “…bound (ELBO) (Zhao et al., 2017) to maximize: where β is a scalar to balance the reconstruction term L REC and the Kullback–Leibler (KL) divergence term L KL and is adjusted by a cyclic annealing schedule (Fu et al., 2019) to alleviate the vanishing latent variable issue (Bowman et al., 2016).”
Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space (2020, methodology): “This annealing procedure can be used once (M.A.) (Bowman et al., 2016) or multiple times (C.A.) (Fu et al., 2019).”
Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning (2019, methodology): “…the cyclical cosine annealing stepsize schedule has been demonstrated to be able to ﬁnd diverse solutions in multimodal objectives, though not speciﬁcally different modes, using stochastic gradient methods (Loshchilov & Hutter, 2016; Huang et al., 2017; Garipov et al., 2018; Fu et al., 2019).”
DLow: Diversifying Latent Flows for Diverse Human Motion Prediction (2020, background): “These methods [68, 46, 6, 61, 42, 71, 3] often build upon popular generative models such as conditional generative adversarial networks (CGANs; [21]) or conditional variational autoencoders (CVAEs; [38]).”
Diffusion-based Generation, Optimization, and Planning in 3D Scenes (2023, methodology): “Although cVAE is easy to train and sample due to its simple architecture and one-step sampling procedure, it suffers from the posterior collapse problem [12, 17, 26, 64, 69, 84, 93]; the learned latent variable is ignored by a strong decoder, leading to limited generation diversity from these collap”
Complementary Auxiliary Classifiers for Label-Conditional Text Generation (2020, background): “Various techniques have been proposed to learn smooth and disentangled latent representations (Higgins et al. 2017; Alemi et al. 2017; Fu et al. 2019).”
Jointly Optimizing Diversity and Relevance in Neural Response Generation (2019, methodology): “Bow-man et al. (2016); Fu et al. (2019) proposed to tackle this problem with word dropping and spe-ciﬁc KL annealing methods.”