Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow

By Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, Sergey Levine. 2018

239 citations recorded by the source corpus.

Adversarial learning methods have been proposed for a wide range of applications, but the training of adversarial models can be notoriously unstable. Effectively balancing the performance of the generator and discriminator is critical, since a discriminator that achieves very high accuracy will produce relatively uninformative gradients. In this work, we propose a simple and general technique to constrain information flow in the discriminator by means of an information bottleneck. By enforcing a constraint on the mutual information between the observations and the discriminator's internal representation, we can effectively modulate the discriminator's accuracy and maintain useful and informative gradients. We demonstrate that our proposed variational discriminator bottleneck (VDB) leads to significant improvements across three distinct application areas for adversarial learning algorithms. Our primary evaluation studies the applicability of the VDB to imitation learning of dynamic continuous control skills, such as running. We show that our method can learn such skills directly from \emph{raw} video demonstrations, substantially outperforming prior adversarial imitation learning methods. The VDB can also be combined with adversarial inverse reinforcement learning to learn parsimonious reward functions that can be transferred and re-optimized in new settings. Finally, we demonstrate that VDB can train GANs more effectively for image generation, improving upon a number of prior stabilization methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Pre-Trained Language Models for Interactive Decision-Making (2022, methodology): “All our experiments use imitation learning (Santara et al., 2017; Ng et al., 2000; Peng et al., 2018), specifically”
InfoMax-GAN: Improved Adversarial Image Generation via Information Maximization and Contrastive Learning (2020, methodology): “Other approaches employing information theoretic principles include Variational GAN (VGAN) [60], which uses an information bottleneck [61] to regularize the discriminator representations; with [62]–[64] extending to minimise divergences apart from the original JS divergence.”
The MAGICAL Benchmark for Robust Imitation (2020, methodology): “As noted in Section 1, several past papers employ “test” variants of standard Gym MuJoCo environments to evaluate IRL generalisation [17, 39, 32, 33], but these modified environments tend to have trivial reward functions (e.”
On Variational Bounds of Mutual Information (2019, background): “These techniques have proven useful in a variety of domains, from restricting the capacity of discriminators in GANs (Peng et al., 2018) to preventing representations from containing information about protected attributes (Moyer et al., 2018).”
Imitation Learning as f-Divergence Minimization (2019, background): “This connection to GANs leads to interesting avenues such as stabilizing min-max games [36], learning from pure observations [37–39] and links to f-divergence minimization [6, 40].”
Task-Relevant Adversarial Imitation Learning (2019, background): “Peng et al. [2018] introduces the Variational Bottleneck to regularize the discriminator.”
Meta-Inverse Reinforcement Learning with Probabilistic Context Variables (2019, background): “Recent works have scaled MaxEnt IRL to large function approximators, such as neural networks, by only partially solving the forward problem in the inner loop, developing an adversarial framework for IRL [7, 8, 11, 25].”
A Survey of Imitation Learning Methods, Environments and Metrics (2024, methodology): “Saliency maps are also used as qualitative metrics in imitation learning literature [38, 79] as model-domain metrics.”
MSG-GAN: Multi-Scale Gradients for Generative Adversarial Networks (2019, background): “between input images and the discriminator’s deepest representation of those input images called the variational discriminator bottleneck (VDB) [25], and Karras et al.”
Positive-Unlabeled Reward Learning (2019, background): “A few works have attempted to address the overfitting problem of GAIL by regularizing the discriminator [10, 11, 12, 20].”
MSG-GAN: Multi-Scale Gradient GAN for Stable Image Synthesis (2019, result): “This is a fundamental issue with GANs, and has been widely reported by previous works [24, 19, 2, 9, 16, 28, 12, 13, 31, 22].”
Using Human Gaze to Improve Robustness Against Irrelevant Objects in Robot Manipulation Tasks (2020, methodology): “versarial optimization [23] has been recently applied to inverse reinforcement learning [24], [25].”
Simple and Effective VAE Training with Calibrated Decoders (2020, unknown): “Peng et al. (2018); Rezende & Viola (2018) further discuss constrained optimization objectives for VAEs, which also yield a similar hyperparameter.”
Information-Bottleneck Approach to Salient Region Discovery (2019, background): “, 2003), generalized dropout (Achille & Soatto, 2018), Generative Adversarial Networks (Peng et al., 2018) and others.”
Behavior-Driven Synthesis of Human Dynamics (2021, methodology): “[45] who constrain discriminator networks, we can optimize (4) using dual gradient decent.”
Imitation Learning: A Survey of Learning Methods, Environments and Metrics (2024, unknown): “Ho and Ermon’s work [4] is the most known work and perhaps one of the most influential, with several methods following the same setting [24, 49, 52, 55].”
Graph Information Bottleneck (2020, unknown): “Other methods apply IB to various domains [40,41].”
Where and How to Transfer: Knowledge Aggregation-Induced Transferability Perception for Unsupervised Domain Adaptation (2021, methodology): “[1], [34] have experimentally shown that the latent representations will be more discriminative 1 if they are closer to a Gaussian distribution.”