How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?

By Ferenc Huszár. 2015

317 citations recorded by the source corpus.

Modern applications and progress in deep learning research have created renewed interest for generative models of text and of images. However, even today it is unclear what objective functions one should use to train and evaluate these models. In this paper we present two contributions. Firstly, we present a critique of scheduled sampling, a state-of-the-art training method that contributed to the winning entry to the MSCOCO image captioning benchmark in 2015. Here we show that despite this impressive empirical performance, the objective function underlying scheduled sampling is improper and leads to an inconsistent learning algorithm. Secondly, we revisit the problems that scheduled sampling was meant to address, and present an alternative interpretation. We argue that maximum likelihood is an inappropriate training objective when the end-goal is to generate natural-looking samples. We go on to derive an ideal objective function to use in this situation instead. We introduce a generalisation of adversarial training, and show how such method can interpolate between maximum likelihood training and our ideal training objective. To our knowledge this is the first theoretical analysis that explains why adversarial training tends to produce samples with higher perceived quality.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Are GANs Created Equal? A Large-Scale Study (2017, background): “For a discussion on other drawbacks of likelihood-based training and evaluation consult Huszár [12].”
Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “…metrics on generated sentences, which show low diversity but high quality for RKL-DPG, compared to other f -DPGs, suggesting it captures a subset of the target distribution (“mode collapse”), as commonly observed in other generative models (Huszar, 2015; Che et al., 2017; Mescheder et al., 2018).”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, background): “, 2022) and the cliff MDP problem (Huszár, 2015; Agarwal et al., 2019; Swamy et al., 2021).”
Wasserstein GAN (2017, background): “GANs oﬀer much more ﬂexibility in the deﬁnition of the objective function, including Jensen-Shannon [4], and all f -divergences [17] as well as some exotic combinations [6].”
f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization (2016, methodology): “In [16] another generalisation of the GAN objective is proposed by using an alternative JensenShannon divergence that interpolates between the KL and the reverse KL divergence and has JensenShannon as its mid-point.”
Assessing Generative Models via Precision and Recall (2018, background): “Other failure modes related to density estimation in highdimensional spaces have been elaborated in [10, 22].”
Towards Principled Methods for Training Generative Adversarial Networks (2017, neutral): “(Theis et al., 2016; Huszar, 2015).”
Texygen: A Benchmarking Platform for Text Generation Models (2018, background): “Notably, since maximum likelihood estimation (MLE) [11] is not a perfect objective function (due to exposure bias explained in [6]), researchers have been looking for improved alternatives.”
Professor Forcing: A New Algorithm for Training Recurrent Networks (2016, background): “However, as remarked by Huszár (2015), scheduled sampling yields a biased estimator, in that even as the number of examples and the capacity go to inﬁnity, this procedure may not converge to the correct model.”
Learning in Implicit Generative Models (2016, methodology): “We often deal with misspeciﬁed models where q θ cannot represent p ∗ , and non maximum likelihood methods could be a more robust choice depending on the task (see ﬁgure 1 in (Huszár, 2015) for an illustrative example).”
Long Text Generation via Adversarial Training with Leaked Information (2017, background): “…so-called exposure bias ∗ Correspondence to Weinan Zhang. due to the discrepancy between training and inference stage: the model sequentially generates the next word based on previously generated words during inference but itself is trained to generate words given ground-truth words (Husz´ar 2015).”
Adversarial Ranking for Language Generation (2017, methodology): “Following this, we take the sentences generated by RankGAN as the input of the oracle model, and estimate the average negative loglikelihood (NLL) [13].”
Improving Abstraction in Text Summarization (2018, background): “Despite its popularity, this approach for sequence generation is suboptimal due to exposure bias (Huszar, 2015) and loss-evaluation mismatch (Wiseman and Rush, 2016).”
Deconvolutional Paragraph Representation Learning (2017, background): “Unfortunately, [20] showed that scheduled sampling is a fundamentally inconsistent ar X iv :1 70 8.”
Pros and Cons of GAN Evaluation Measures (2018, background): “For further discussions on other drawbacks of average likelihood measures consult [37].”
Autoregressive Knowledge Distillation through Imitation Learning (2020, methodology): “For example, directly reusing the tokens in the static dataset as the target next token leads to an inconsistent training procedure (Huszár, 2015).”