Connecting Generative Adversarial Networks and Actor-Critic Methods

By David Pfau, Oriol Vinyals. 2016

197 citations recorded by the source corpus.

Both generative adversarial networks (GAN) in unsupervised learning and actor-critic methods in reinforcement learning (RL) have gained a reputation for being difficult to optimize. Practitioners in both fields have amassed a large number of strategies to mitigate these instabilities and improve training. Here we show that GANs can be viewed as actor-critic methods in an environment where the actor cannot affect the reward. We review the strategies for stabilizing training for each class of models, both those that generalize between the two and those that are particular to that model. We also review a number of extensions to GANs and RL algorithms with even more complicated information flow. We hope that by highlighting this formal connection we will encourage both GAN and RL communities to develop general, scalable, and stable algorithms for multilevel optimization with deep networks, and to draw inspiration across communities.

Subfields: Artificial Intelligence, Reinforcement Learning, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Generative Models

Citations in the corpus

Stochastic Gradient Descent-Ascent and Consensus Optimization for Smooth Games: Convergence Analysis under Expected Co-coercivity (2021, background): “Such ﬁnite sum formulations appear in several machine learning applications such as generative adversarial networks (GANs) [Goodfellow et al., 2014], robust learning [Wen et al., 2014] or even some formulations of reinforcement learning [Pfau and Vinyals, 2016].”
Stochastic Hamiltonian Gradient Methods for Smooth Games (2020, background): “Indeed, in problems like domain generalization (Albuquerque et al., 2019), generative adversarial networks (Goodfellow et al., 2014), and some formulations in reinforcement learning (Pfau & Vinyals, 2016), empirical risk minimization yields ﬁnite sums of the form of (3).”
A Tight and Unified Analysis of Extragradient for a Whole Spectrum of Differentiable Games (2019, background): “For instance, the actor-critic model (Pfau and Vinyals, 2016), generative adversarial networks (GANs) (Goodfellow et al., 2014) and automatic curricula (Sukhbaatar et al., 2018) can be cast as two-player games.”
Negative Momentum for Improved Game Dynamics (2018, methodology): “For example, actor-critic methods can be written as a bi-level optimization problem (Pfau and Vinyals, 2016) and generative adversarial networks (GANs) (Goodfellow et al.”
The Numerics of GANs (2017, background): “For example, the popular actor-critic models [20] in reinforcement learning are also special cases of saddle-point problems.”
NIPS 2016 Tutorial: Generative Adversarial Networks (2016, background): “Researchers have already identified connections between GANs and actor-critic methods (Pfau and Vinyals, 2016), inverse reinforcement learning (Finn et al.”
Recent Progress on Generative Adversarial Networks (GANs): A Survey (2019, background): “For instance, [70] embedded GANs into imitation learning; [61] combined GANs with policy gradient; [71] described the relationship between GANs and actor-critic algorithms, etc.”
The Mechanics of n-Player Differentiable Games (2018, background): “Copyright 2018 by the author(s). optimization (Pfau & Vinyals, 2016), synthetic gradients (Jaderberg et al., 2017), hierarchical reinforcement learning (Wayne & Abbott, 2014; Vezhnevets et al., 2017), curiosity (Pathak et al., 2017), and imaginative agents (Racani`ere et al., 2017).”
An Online Learning Approach to Generative Adversarial Networks (2017, background): “…works in the parameter space - and many issues arise during training Arjovsky and Bottou (2017); Radford et al. (2015), thus requiring careful initialization and proper regularization as well as other tricks Metz et al. (2016); Pfau and Vinyals (2016); Radford et al. (2015); Salimans et al. (2016).”
Interaction Matters: A Note on Non-asymptotic Local Convergence of Generative Adversarial Networks (2018, background): “Optimization of GANs (and value functions of the form (1.1) at large) is hard, both in theory and in practice [Singh et al., 2000, Pfau and Vinyals, 2016, Salimans et al., 2016].”
TF-Replicator: Distributed Machine Learning for Researchers (2019, background): “…TensorFlow’s release, multi-loss methods have proliferated, including proximal gradient TD-learning (Liu et al., 2016), multi-level optimization (Pfau & Vinyals, 2016), synthetic gradients (Jaderberg et al., 2016) and RL agents that interact with multiple losses in increasingly complex ways…”
Recent advances in deep learning based dialogue systems: a systematic survey (2021, background): “It is easy to link the actor-critic model with another framework - GANs (Goodfellow et al., 2014) because of their similar inner structure and logic (Pfau and Vinyals, 2016).”
DeepSMOTE: Fusing Deep Learning and SMOTE for Imbalanced Data (2021, background): “GANs formulate image generation as a min–max game between a generator and a discriminator network [72].”
Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets (2017, background): “More recently, there have been works connecting GANs and other reinforcement learning and IRL methods [24, 11, 16].”
Accelerating Smooth Games by Manipulating Spectral Shapes (2020, background): “The recent successes of multi-agent formulations in various areas of deep learning (Goodfellow et al., 2014; Pfau and Vinyals, 2016) have caused a surge of interest in the theoretical understanding of first-order methods for the solution of differentiable multi-player games (Balduzzi et al.”
Differentiable Game Mechanics (2019, background): “…adversarial networks (Goodfellow et al., 2014; Zhu et al., 2017), proximal gradient TD learning (Liu et al., 2016), multi-level optimization (Pfau and Vinyals, 2016), synthetic gradients (Jaderberg et al., 2017), hierarchical reinforcement learning (Wayne and Abbott, 2014; Vezhnevets et…”
Generative Adversarial Parallelization (2016, background): “Recent work has explored the connection between GANs and actor-critic methods in reinforcement learning (Pfau & Vinyals, 2016).”
Adversarial Transfer Learning (2018, background): “GANs can also be viewed as a type of actor critic approach to reinforcement learning [98].”